Category Archives: Linux

Setting up a 2-node GlusterFS filesystem

This will be a quick howto on how you would set up a 2-node GlusterFS filesystem. You may look up more information at http://www.gluster.org/.

Volume types for GlusterFS

– Distributed. Distributed volumes distributes files throughout the bricks in the volume
– Replicated. Replicated volumes replicates files across bricks in the volume
– Striped. Striped volumes stripes data across bricks in the volume
– Distributed Striped. Distributed striped volumes stripe data across two or more nodes in the cluster
– Distributed Replicated. Distributed replicated volumes distributes files across replicated bricks in the volume
– Distributed Striped Replicated. Distributed striped replicated volumes distributes striped data across replicated bricks in the cluster
– Striped Replicated. Striped replicated volumes stripes data across replicated bricks in the cluster

The high level overview of how the process will be is as follows

  • Installing the required software
  • Disable or add proper firewall rules
  • Adding nodes into the cluster
  • Preparing “bricks” for use on each server
  • Creating and starting the actual GlusterFS volume
  • Mounting the GlusterFS volume
  • Installing the required software

    I will be providing examples for CentOS, Fedora, Debian and Arch Linux. The examples for CentOS will work for RHEL and Scientific Linux as well.
    CentOS
    The following command will install all dependencies.

    # yum install glusterfs

    Fedora
    The following command will install all dependencies.

    # yum install glusterfs-server

    Debian
    The following command will install all dependencies.

    # apt-get install glusterfs-server

    Arch Linux
    The following command will install all dependencies.

    # pacman -S glusterfs

    Disable or add proper firewall rules

    You will need to open the following ports for GlusterFS.

    24007 – GlusterFS Daemon
    24008 – Management
    24009 - Each brick for every volume on your host requires it’s own port. For every new brick, one new port will be used starting at 24009. (For GlusterFS versions earlier than 3.4)
    49152 - Each brick for every volume on your host requires it’s own port. For every new brick, one new port will be used starting at 49152 (GlusterFS 3.4 and later)
    38465:38467 - This is required if you use the GlusterFS NFS service.
    

    CentOS
    Disabling the default firewall

    # chkconfig iptables off
    # service stop iptables

    Fedora

    systemctl disable firewalld
    systemctl stop firewalld

    Debian
    There are no default firewall installed on Debian.
    Arch Linux
    There are no default firewall installed on Arch Linux.

    Adding nodes into the cluster

    This is incredibly easy. You may do the following command from either server. In my example I am on server1. If you don’t have a solid DNS you should add each server to each others hosts file.

    # gluster peer probe server2
    Probe successful

    Preparing “bricks” for use on each server

    Nothing fanzy, you just need to create folders. It’s also important to note that you will need to use a folder, even if you intended to use a single disk.
    Execute the following on both of your servers

    # mkdir -p /data/brick>

    Creating and starting the actual GlusterFS volume

    Creating the GlusterFS volume
    Syntax:

    gluster volume create NEW-VOLNAME [replica COUNT] [transport [tcp | rdma | tcp,rdma]] NEW-BRICK...

    Example:

    # gluster volume create test-volume replica 2 transport tcp server1:/data/brick server2:/data/brick
    Creation of test-volume has been successful
    Please start the volume to access data.
    

    Starting the GlusterFS volume

    # gluster volume start test-volume

    Mounting the GlusterFS volume

    It’s important to note that you will need to mount the GlusterFS to use it. WARNING: Adding files directly to a brick will not be included in a GlusterFS volume.
    Syntax:

    # mount.glusterfs servername:volumename /mnt/mountpoint

    Examples:

    # mount.glusterfs server1:test-volume /mnt/glusterfs/

    OR

    # mount -t glusterfs server1:test-volume /mnt/glusterfs/

    References

    http://www.gluster.org/wp-content/uploads/2012/05/Gluster_File_System-3.3.0-Administration_Guide-en-US.pdf
    http://gluster.org/community/documentation/index.php/QuickStart

    // CrashMAG

    How to identify ECC memory modules

    This is a short article describing how you proceed to identify whether or not you have ECC memory modules in your Linux workstation or server.

    Also as a side note, the importance of ECC memory is great. Even filesystems such as ZFS with check summing will not account for flipped bits due to cosmic rays. According to studies such as http://www.cs.toronto.edu/~bianca/papers/sigmetrics09.pdf , a DIMM has an 8% chance per year of getting a correctable error. Multiply that with the amount of DIMM’s you have in your system (4 or more?), and you suddenly have a very likely chance of seeing data corruption during a year.

    To display what type of memory module you have, we make use of the following DMI type:

    16   Physical Memory Array

    Command

    # dmidecode --type 16

    Output

    # dmidecode 2.11
    SMBIOS 2.7 present.
    
    Handle 0x0007, DMI type 16, 23 bytes
    Physical Memory Array
            Location: System Board Or Motherboard
            Use: System Memory
            Error Correction Type: Single-bit ECC
            Maximum Capacity: 32 GB
            Error Information Handle: 0x0010
            Number Of Devices: 4
    

    Both on Debian/Ubuntu and RedHat based distributions this tool is provided by the dmidecode package.

    // CrashMAG

    How to configure network bonding (LACP) on Debian Wheezy

    This process essentially consist of two steps. I will be detailing steps relevant for the Linux host.

    • Configuring the switch for LACP bonding.
    • Configuring the Linux host for LACP bonding.

    Prerequisites

    • ifenslave
    • Shut down the network after installing ifenslave.
    • Start the network once the configuration changes are in place.

    Steps

    This is a virtual package and will in reality install ifenslave-2.6

    # aptitude install ifenslave

    Stop the network. Make sure you’re not connected via SSH while doing this.

    # /etc/init.d/networking stop

    Debian Kernel Module Configuration

    File: /etc/modprobe.d/bonding.conf

    #/etc/modprobe.d/bonding.conf
    alias bond0 bonding
            options bonding mode=4 miimon=100 lacp_rate=1

    File: /etc/modules

    echo "bonding" >> /etc/modules
    echo "mii" >> /etc/modules

    Debian Network Configuration

    /etc/network/interfaces
    #/etc/network/interfaces 
    auto eth0
        iface eth0 inet manual
        bond-master bond0
    
    auto eth1
         iface eth1 inet manual
         bond-master bond0
    
    auto bond0
         iface bond0 inet static
         address 192.168.0.10
         gateway 192.168.0.1
         netmask 255.255.255.0
         bond-mode 802.3ad
         bond-miimon 100
         bond-downdelay 200
         bond-updelay 200
         bond-lacp-rate 4
         bond-slaves none
         dns-nameservers 192.168.0.1
         dns-search domain.int

    Start up the network.

    # /etc/init.d/networking start

    // CrashMAG

    Linux KVM host to guest connectivity

    If you’re experience a lack of connectivity between your KVM host and your guests please see below. The instructions below will only directly work on Debian and/or Ubuntu. They will also require your guests to use macvlan or macvtap. This will also work if you’re using LXC.

    Add the following to your

    /etc/network/interfaces

    configuration file. You need to adjust the network portion of the example below according to your own setup.

    auto macvlan0
    iface macvlan0 inet dhcp
        # as eth0 and macvlan0 are on the same LAN, we must drop default route and LAN route
        pre-up route del default
        pre-up route del -net 192.168.0.0 netmask 255.255.255.0
        pre-up ip link add link eth0 name macvlan0 type macvlan mode bridge
    

    Now, either reboot or run

    ifup macvlan0

    as root.

    // CrashMAG

    Disable the filesystem check (fsck) at boot time

    There’s several ways of accomplishing this. I will list all the methods beneath, just pick the one that fits the situation/you.

    • Filesystem tunable
    • Grub boot parameter
    • Placing command files on your root device
    • Active reboot without FSCK

    Filesystem tunable

    Use the tune2fs command to tell your filesystem to have a max count of mounts before a check to 0 to disable it.

    # tune2fs -c 0 /dev/sda1

    Parameter reference:

    -c max-mount-counts
     Adjust the number of mounts after which the filesystem will be  checked  by  e2fsck(8).   If max-mount-counts  is  0  or -1, the number of times the filesystem is mounted will be disregarded by e2fsck(8) and the kernel.
    

    Grub boot parameter

    Add the following at the end of your grub boot linux line.

    fastboot

    This can be done by editing “grub.conf” or by editing the boot command via the grub menu at boot.

    Placing command files on your root device

    To disable the filesystem check on boot.

    # touch /fastboot

    To enable a filesystem check on boot.

    # touch /forcefsck

    Active reboot without FSCK

    # shutdown -rf

    Parameter reference:

    -r     Reboot after shutdown.
    -f     Skip fsck on reboot.
    

    // CrashMAG

    Testing SMTP, POP3 and IMAP protocol access

    This article assumes you have access to telnet and openssl. The example tests have been run against a Microsoft Exchange 2010 server. The IP and hostname have been obfuscated. The commands needed to perform these protocol access tests will be the same on both Linux and Windows.

    Testing SMTP

    Test using plain text

    Execute the following command to initiate a plain text connection over port 25.

    telnet smtp.server.com 25

    Example output

    The following is the typical output you’ll see as a response from a SMTP server. In this case being Microsoft Exchange 2010.

    Trying 74.161.5.111...
    Connected to smtp.server.com.
    Escape character is '^]'.
    220 smtp.server.com Microsoft ESMTP MAIL Service ready at Thu, 3 May 2012 13:06:21 +0200
    

    Test using an encrypted connection

    Execute the following command to initiate an encrypted connection over port 25.

    openssl s_client -starttls smtp -crlf -connect smtp.server.com:25

    Parameters

    Beneath you’ll see the documentation for the parameters used in the above example.

    -starttls protocol
    send the protocol-specific message(s) to switch to TLS for communication.  protocol is a keyword for the intended protocol.  Currently, the only supported keywords are "smtp", "pop3", "imap", and "ftp".
    
    -crlf
    this option translated a line feed from the terminal into CR+LF as required by some servers.
    

    Example output

    There’s little to see here mainly because I had to exclude the certificate verification information to anonymize the test server.

    <certificate verification output>
    250 CHUNKING
    

    Tip: You may run the usual SMTP commands directly from the command prompt after you initiated the encrypted connection.

    Testing IMAP

    Test using plain text

    Execute the following command to initiate a plain text connection over the standard IMAP port 143.

    telnet imap.server.com 143

    Example output

    The following is the typical output you’ll see as a response from an IMAP server. In this case being Microsoft Exchange 2010.

    Trying 74.161.5.111...
    Connected to imap.server.com.
    Escape character is '^]'.
    * OK The Microsoft Exchange IMAP4 service is ready.
    

    Test using an encrypted connection

    openssl s_client -connect imap.server.com:993

    Example output

    <certificate verification output>
    * OK The Microsoft Exchange IMAP4 service is ready.
    

    Testing POP3

    Test using plain text

    telnet pop.server.com 110

    Example output

    The following is the typical output you’ll see as a response from a POP server. In this case being Microsoft Exchange 2010.

    Trying 74.161.5.111...
    Connected to pop.server.com.
    Escape character is '^]'.
    +OK The Microsoft Exchange POP3 service is ready.
    

    Test using an encrypted connection

    openssl s_client -connect pop.server.com:995

    Example output

    <certificate verification output>
    +OK The Microsoft Exchange POP3 service is ready.
    

    References

    SMTP – Simple Mail Transfer Protocol
    IMAP – INTERNET MESSAGE ACCESS PROTOCOL
    POP 3 – Post Office Protocol – Version 3
    The OpenSSL Project

    // CrashMAG

    Linux ACL

    An access control list (ACL), with respect to a computer file system, is a list of permissions attached to an object. ACL allows you to grant or deny permissions for any user or group on a filesystem resource.

    Enabling ACL

    To enable ACL, edit your /etc/fstab file as such:

    /dev/VolGroup00/LogVol00 /                       ext3    defaults,acl        1 1

    Note: Moderm Redhat distributions enable ACL by default for the root filesystem.

    Set ACL

    To modify ACL use setfacl command. To add permissions use setfacl -m.

    Add permissions to some user:

    # setfacl -m "u:username:permissions"

    or

    # setfacl -m "u:uid:permissions"

    Add permissions to some group:

    # setfacl -m "g:groupname:permissions"

    or

    # setfacl -m "g:gid:permissions"

    Add default ACL:

    # setfacl -d -m "u:uid:permissions"

    Remove all permissions:

    # setfacl -b

    Remove each entry:

    # setfacl -x "entry"

    To check permissions use:

    # getfacl filename

    Examples

    Set read,write and execute permissions for user “johndoe” on the file named “abc”.

    # setfacl -m "u:johndoe:rwx" abc

    Check permissions.

    # getfacl abc
    # file: abc
    # owner: someone
    # group: someone
    user::rw-
    user:johny:rwx
    group::r--
    mask::rwx
    other::r--

    Change permissions for user “johndoe”.

    # setfacl -m "u:johndoe:rw-" abc

    Check permissions.

    # getfacl abc
    # file: abc
    # owner: someone
    # group: someone
    user::rw-
    user:johndoe:rw-
    group::r--
    mask::r-x
    other::r--

    Remove all extended ACL entries.

    # setfacl -b abc

    Check permissions.

    # getfacl abc
    # file: abc
    # owner: someone
    # group: someone
    user::rw-
    group::r--
    other::r--

    Additional Resources

    man getfacl
    man setfacl

    If you weren’t using these already, you should.

    // CrashMAG

    Making sure that Phusion Passenger (Mod_rails) is able to run Chiliproject

    To be able to run Chiliproject (Fork of Redmine) using Phusion Passenger you need to implement 1 work around. It doesn’t seem like Passenger is aware of environment variables. I saw the following errors in “log/production.log”

    Processing ApplicationController#index (for 192.168.0.105 at 2011-08-14 13:40:01) [GET]
      Parameters: {"controller"=>"settings", "action"=>"index"}
    
    ArgumentError (invalid byte sequence in US-ASCII):
      :10:in `synchronize'
      :10:in `synchronize'
    
    Rendering /srv/http/chiliproject/public/500.html (500 Internal Server Error)

    Therefore create the following executeable using the code beneath and save it as e.g. /usr/bin/ruby-passenger.

    #!/bin/bash
    exec /usr/bin/ruby -E utf-8:utf-8 "$@"
    

    Make sure you make it executable by running

    # chmod +x /usr/bin/ruby-passenger

    // CrashMAG

    Useful GNU/Linux search commands

    These will work on any GNU/Linux system.

    Find the email address someone@example.com within the path /etc recursively

    grep -H -r "someone@example.com" /etc

    -H, –with-filename
    Print the file name for each match.
    -R, -r, –recursive
    Read all files under each directory, recursively

    Find every file under the directory /home owned by the user john

    find /home -user john

    Find every file under the directory /usr ending in ”log”

    find /usr -name *log

    Find every file under the directory /etc that was modified more than 60 days ago

    find /etc -mtime +60

    Runs `file’ on every file in or below the current directory

    find . -type f -exec file '{}' \;

    Search for files in your home directory which have been modified in the last twenty-four hours. This command works this way because the time since each file was last modified is divided by 24 hours and any remainder is discarded. That means that to match -mtime

    find $HOME -mtime 0

    Search for files which have read and write permission for their owner, and group, but which other users can read but not write to. Files which meet these criteria but have other permissions bits set (for example if someone can execute the file) will not be matched

    find . -perm 664

    Search for files which have read and write permission for their owner and group, and which other users can read, without regard to the presence of any extra permission bits (for example the executable bit)

    find . -perm -664

    Search for files which are writable by somebody (their owner, or their group, or anybody else)

    find . -perm /222

    All three of these commands do the same thing, but the first one uses the octal representation of the file mode, and the other two use the symbolic form. These commands all search for files which are writable by either their owner or their group. The files don’t have to be writable by both the owner and group to be matched; either will do

    find . -perm /220
    find . -perm /u+w,g+w
    find . -perm /u=w,g=w
    

    Both these commands do the same thing; search for files which are writable by both their owner and their group

    find . -perm -220
    find . -perm -g+w,u+w
    

    These two commands both search for files that are readable for everybody (-perm -444 or -perm -a+r), have at least on write bit set (-perm /222 or -perm /a+w) but are not executable for anybody (! -perm /111 and ! -perm /a+x respectively)

    find . -perm -444 -perm /222 ! -perm /111
    find . -perm -a+r -perm /a+w ! -perm /a+x
    

    // CrashMAG

    Change the default SSH port and alter SELinux context to match

    Security through obscurity is not something one would generally recommend. But to thwart the effort of automated scanners changing the default OpenSSH port will yield you less pain in every day life. This will not fend off directed attacks or nullify vulnerabilities or bad security design.

    Should you see an error message such as

    shd[14221]: error: Bind to port 9898 on 192.168.0.50 failed: Permission denied

    it indicates that the system prevented the daemon to bind that port. Most likely SELinux.

    The instructions provided will be valid on Fedora 14/15, CentOS 6, RHEL 6, Scientific Linux 6 and newer versions.

    To change the default SSH port you need to do the following.

    • Stop the SSH daemon
    • Alter the /etc/ssh/sshd_config with your new port
    • Alter the SELinux context with semanage
    • Start the SSH daemon

    Stop the SSH daemon

    # service sshd stop

    Alter the /etc/ssh/sshd_config with your new port

    Alter the configuration file with your favorite editor, in my case “nano”.

    # nano /etc/ssh/sshd_config

    Alter the port configuration parameter change the following line

    Port 22

    to

    Port 9898

    Alter the SELinux context with semanage

    # semanage port -a -t ssh_port_t -p tcp 9898

    Initially you would think the following would work. But it will not. For it to work you would have to alter the policy in the selinux-policy package, rebuild and install it. So skip it, but now you know why.

    # semanage port -d -t ssh_port_t -p tcp 22

    Start the SSH daemon

    # service sshd start

    // CrashMAG