QNAP TS-212 How to rebuild RAID manually from telnet

Scenario = replace a disk in QNAP TS-212 with RAID 1 configuration active

RAID rebuild should start automatically, but some times it could happen you got stuck with 1 Single Disk + 1 Mirroring Disk Volume:

SingleDisk

According to the QNAP Support – How can I migrate from Single Disk to RAID 0/1 in TS-210/TS-212? , TS-210/TS-212 does not support Online RAID Level Migration. Therefore, please backup the data on the single disk to another location, install the second hard drive, and then recreate the new RAID 0/1 array (hard drive must be formatted).

The workaround is this:

  1. Telnet to NAS as Admin
  2. Check your current disk configuration for Disk #1 and Disk #2 =
    fdisk -l /dev/sda

    Disk /dev/sdb: 1000.2 GB, 1000204886016 bytes
    255 heads, 63 sectors/track, 121601 cylinders
    Units = cylinders of 16065 * 512 = 8225280 bytes

       Device Boot      Start         End      Blocks   Id  System
    /dev/sdb1               1          66      530125   83  Linux
    /dev/sdb2              67         132      530142   83  Linux
    /dev/sdb3             133      121538   975193693   83  Linux
    /dev/sdb4          121539      121600      498012   83  Linux

    fdisk -l /dev/sdb

    Disk /dev/sda: 1000.2 GB, 1000204886016 bytes
    255 heads, 63 sectors/track, 121601 cylinders
    Units = cylinders of 16065 * 512 = 8225280 bytes

       Device Boot      Start         End      Blocks   Id  System
    /dev/sda1               1          66      530125   83  Linux
    /dev/sda2              67         132      530142   83  Linux
    /dev/sda3             133      121538   975193693   83  Linux
    /dev/sda4          121539      121600      498012   83  Linux

  3. SDA is the first disk, SDB is the second disk
  4. Verify the current status of RAID with this command =
    mdadm –detail /dev/md0

    /dev/md0:
            Version : 00.90.03
      Creation Time : Thu Sep 22 21:50:34 2011
         Raid Level : raid1
         Array Size : 486817600 (464.27 GiB 498.50 GB)
      Used Dev Size : 486817600 (464.27 GiB 498.50 GB)
       Raid Devices : 2
      Total Devices : 1
    Preferred Minor : 0
        Persistence : Superblock is persistent

      Intent Bitmap : Internal

        Update Time : Thu Jul 19 01:13:58 2012
              State : active, degraded
    Active Devices : 1
    Working Devices : 1
    Failed Devices : 0
      Spare Devices : 0

               UUID : 72cc06ac:570e3bf8:427adef1:e13f1b03
             Events : 0.1879365

        Number   Major   Minor   RaidDevice State
           0       0        0        0      removed
           1       8        3        1      active sync  /dev/sda3

  5. As you can see the /dev/sda3 is working, so disk #1 is OK, but disk #2 is missing from RAID
  6. Check if Disk #2 /dev/sdb is mounted (it should be) =
    mount

    /proc on /proc type proc (rw)
    none on /dev/pts type devpts (rw,gid=5,mode=620)
    sysfs on /sys type sysfs (rw)
    tmpfs on /tmp type tmpfs (rw,size=32M)
    none on /proc/bus/usb type usbfs (rw)
    /dev/sda4 on /mnt/ext type ext3 (rw)
    /dev/md9 on /mnt/HDA_ROOT type ext3 (rw)
    /dev/md0 on /share/MD0_DATA type ext4 (rw,usrjquota=aquota.user,jqfmt=vfsv0,user_xattr,data=ordered,delalloc,noacl)
    tmpfs on /var/syslog_maildir type tmpfs (rw,size=8M)
    /dev/sdt1 on /share/external/sdt1 type ufsd (rw,iocharset=utf8,dmask=0000,fmask=0111,force)
    tmpfs on /.eaccelerator.tmp type tmpfs (rw,size=32M)
    /dev/sdb3 on /share/HDB_DATA type ext3 (rw,usrjquota=aquota.user,jqfmt=vfsv0,user_xattr,data=ordered,noacl)

  7. Dismount the /dev/sdb3 Disk #2 with this command =
    umount /dev/sdb3
  8. Add Disk #2 into the RAID /dev/md0 =
    mdadm /dev/md0 –add /dev/sdb3

    mdadm: added /dev/sdb3

  9. Check the RAID status and the rebuild should be started automatically =
    mdadm –detail /dev/md0

    /dev/md0:
            Version : 00.90.03
      Creation Time : Thu Sep 22 21:50:34 2011
         Raid Level : raid1
         Array Size : 486817600 (464.27 GiB 498.50 GB)
      Used Dev Size : 486817600 (464.27 GiB 498.50 GB)
       Raid Devices : 2
      Total Devices : 2
    Preferred Minor : 0
        Persistence : Superblock is persistent

      Intent Bitmap : Internal

        Update Time : Thu Jul 19 01:30:27 2012
              State : active, degraded, recovering
    Active Devices : 1
    Working Devices : 2
    Failed Devices : 0
      Spare Devices : 1

    Rebuild Status : 0% complete

               UUID : 72cc06ac:570e3bf8:427adef1:e13f1b03
             Events : 0.1879848

        Number   Major   Minor   RaidDevice State
           2       8       19        0      spare rebuilding   /dev/sdb3
           1       8        3        1      active sync   /dev/sda3

  10. Check the NAS site for the rebuild % progress

    image

  11. After the RAID rebuild complete, restart NAS to clean all previous mount point folder for sdb3

28 thoughts on “QNAP TS-212 How to rebuild RAID manually from telnet

  1. Thank you for sharing this. I should point that i tried that on my 410 but mdadm command should start by — and not a single one.

    I’m having some trouble trying to replace a failed disk in my raid5 with one of higher capacity (because i haven’t one of same size) rebuild doesn’t start automatically and when i follow your steps i realized there is no sda3 so i’m assuming disk hasn’t been formatted:

    [~] # fdisk -l /dev/sda
    You must set cylinders.
    You can do this from the extra functions menu.

    Disk /dev/sda: 0 MB, 0 bytes
    255 heads, 63 sectors/track, 0 cylinders
    Units = cylinders of 16065 * 512 = 8225280 bytes

    Device Boot Start End Blocks Id System
    /dev/sda1 1 66 530125 83 Linux
    /dev/sda2 67 132 530142 83 Linux

    Please. How can i format the disk, so i can add it to the raid?

  2. After the rebuild and restart of the NAS the “Current Configuration : Logical Volumes” still shows up wrong. Checking via “mdadm –detail /dev/md0” there was gain only one drive active… I will give it a 2nd try. In order to speed up the rebuild I have stopped all service via “/etc/init.d/services.sh stop” – this was speeding up the rebuild by a factor of between 10x and 20x. You can monitor the process via “cat /proc/mdstat”:

    ~] # cat /proc/mdstat
    Personalities : [linear] [raid0] [raid1] [raid10] [raid6] [raid5] [raid4] [multipath]
    md0 : active raid1 sdb3[2] sda3[0]
    1463569600 blocks [2/1] [U_]
    [>………………..] recovery = 2.4% (35956480/1463569600) finish=324.6min speed=73282K/sec
    bitmap: 3/175 pages [12KB], 4096KB chunk

  3. Thanks. It looks like that did the trick. It’s rebuilding right now.

    one remark.

    mdadm –detail /dev/md0

    needed to be:

    mdadm –detail /dev/md0

    and

    mdadm /dev/md0 –add /dev/sdb3

    needed to be:

    mdadm /dev/md0 -–add /dev/sdb3

  4. Pingback: QNAP TS-212 How to rebuild RAID manually from telnet - ROBIT Sistemas Informaticos y Comunicaciones, S.L | ROBIT Sistemas Informaticos y Comunicaciones, S.L

  5. Pingback: QNAP TS-212 How to rebuild RAID manually from telnet - Roberto Martinez Martin - Web Personal | Roberto Martinez Martin – Web Personal

  6. Pingback: Raid Recovery Failed Qnap

  7. Pingback: Qnap Raid Recovery Mdadm

  8. Pingback: Qnap Raid Recovery Ssh

  9. I’m so glad you published this. I had this exact model NAS with this exact problem and it was solved with your steps. I was really getting worried that I wasn’t going to my new disk to work. Whew!

  10. This saved my skin (I think!) – could you post up a screenshot of what volume management looks like after mirroring is complete? My disk #2 still says “single disk drive 2”, whereas number 1 says “Mirroring disk volume drive 1”. Is this correct? The free size is the same on both so I’m guessing it’s ok….

      • Hi Ing – I meant after the mirroring has completed? Mine still says “single disk drive 2” and is not continuing to mirror. For example, if I delete say a 1Gb file, it only removes the 1Gb from disk #1 and is not mirrored by disk 2. Do you have any ideas what could cause this? Thanks for your help!🙂

      • After the rebuild you should not have Single Disk anymore and you should see only a single RAID disk like before. Did you restart your NAS at end of rebuild process?

        Something like this:

      • ok I repeated your instructions and it works perfectly until the final stage with Raid 1 working. However after the reboot it goes back to degraded mode with sdb3 missing from the Raid…. help!

  11. Hi Ing – after the rebuild I have an unmounted single disk 2. Is this wrong? On the ssh terminal using “mdadm –detail /dev/md0”, it appears that everything is ok – status is ‘active’. When I reboot the NAS, disk 2 then gets removed from the RAID again and the same problem happens. Any other ideas?

  12. Hi Marc,
    I faced the problem once… it was because the RAID replicate the data content but the boot partition is only on disk 1.
    Let say we have this scenario
    * disk 1 is broken and replaced with new disk
    * disk 2 is still active

    First time you turn on the NAS, it will boot from disk 2 as disk 1 is empty.
    it will rebuild RAID and copy data on disk 1 but it still using disk 2
    after you reboot, disk 1 will be used to boot instead of disk 2 and that break your raid information.

    Try to do this:
    * Rebuild the RAID to 100%
    * Now, both disk should have all data replicated
    * Shutdown the NAS
    * Unplug the old disk (2 in example above) which was used to boot as it was the only working one.
    * switch new disk (1) to position of old disk (2) so it will be used to boot
    * turn on the NAS and see if it boot correctly
    * check if you have all your data (it should because RAID was rebuilt to 100%)
    * shutdown the NAS
    * connect old disk (2) to the free slot (used previously by disk 1)
    * start NAS and check if all data is there and status or RAID
    * you may need to rebuild RAID once again on old disk (2)

    Summary is:
    * NAS usually boot only from disk 1
    * You need to slip disk 1 and disk 2 to be able to rebuild RAID correctly.

    Hope this help!

  13. Thanks for everyone’s help. In the end we got Qnap to log in and the very helpful Arun sorted it out. For anyone who’s interested, here are the ssh commands he used:

    [admin@Holly ~]# mount
    /proc on /proc type proc (rw)
    none on /dev/pts type devpts (rw,gid=5,mode=620)
    sysfs on /sys type sysfs (rw)
    tmpfs on /tmp type tmpfs (rw,size=32M)
    none on /proc/bus/usb type usbfs (rw)
    /dev/sda4 on /mnt/ext type ext3 (rw)
    /dev/md9 on /mnt/HDA_ROOT type ext3 (rw,data=ordered)
    /dev/sdb3 on /share/HDB_DATA type ext3 (rw,usrjquota=aquota.user,jqfmt=vfsv0,user_xattr,data=ordered,noacl)
    /dev/md0 on /share/MD0_DATA type ext3 (rw,usrjquota=aquota.user,jqfmt=vfsv0,user_xattr,data=ordered,noacl)
    tmpfs on /mnt/rf/nd type tmpfs (rw,size=1m)
    [admin@Holly ~]#
    [admin@Holly ~]#
    [admin@Holly ~]#
    [admin@Holly ~]#
    [admin@Holly ~]# cat /etc/config/raidtab
    raiddev /dev/md0
    raid-level 1
    nr-raid-disks 2
    nr-spare-disks 0
    chunk-size 4
    persistent-superblock 1
    device /dev/sda3
    raid-disk 0
    device
    raid-disk 1
    [admin@Holly ~]#
    [admin@Holly ~]# head /etc/config/uLinux.conf
    [System]
    Model = TS-210
    Internal Model = TS-219
    Server comment =
    Version = 4.2.0
    Build Number = 20151118
    Time Zone = Europe/London
    Enable Daylight Saving Time = TRUE
    Workgroup = home
    Code Page = 437
    [admin@Holly ~]#
    [admin@Holly ~]#
    [admin@Holly ~]#
    [admin@Holly ~]# vi /etc/config/raidtab
    raiddev /dev/md0
    raid-level 1
    nr-raid-disks 2
    nr-spare-disks 0
    chunk-size 4
    persistent-superblock 1
    device /dev/sda3
    raid-disk 0
    device /dev/sdb3
    raid-disk 1
    ~
    ~
    ~
    ~
    “/mnt/HDA_ROOT/.config/raidtab” 10 lines, 169 characters written
    [admin@Holly ~]#
    [admin@Holly ~]# /etc/init.d/smb
    smb.sh smb2_protocol.sh
    [admin@Holly ~]#
    [admin@Holly ~]#
    [admin@Holly ~]# /etc/init.d/services.sh restart
    Stop qpkg service: /etc/rcK.d/QK100tvmobili: line 7: /opt/var/lib/dbus/machine-id: No such file or directory
    Stopping
    Shutting down UPnP MediaServer services: twonkymedia.
    Shutting down Download Station services:
    .
    Stop service: vpn_openvpn.sh vpn_pptp.sh ldap_server.sh antivirus.sh iso_mount.sh qbox.sh qsyncman.sh rsyslog.sh snmp lunportman.sh iscsitrgt.sh twonkymedia.sh init_iTune.sh ImRd.sh crond.sh nvrd.sh StartMediaService.sh mariadb.sh btd.sh mysqld.sh recycled.sh Qthttpd.sh atalk.sh nfs http://ftp.sh smb.sh versiond.sh .
    Start services: smb.sh http://ftp.sh nfs atalk.sh Qthttpd.sh recycled.sh mysqld.sh btd.sh mariadb.sh StartMediaService.sh nvrd.sh crond.sh ImRd.sh init_iTune.sh twonkymedia.sh lunportman.sh iscsitrgt.sh snmp rsyslog.sh qsyncman.sh qbox.sh iso_mount.sh ldap_server.sh vpn_pptp.sh vpn_openvpn.sh versiond.sh antivirus.sh .
    Start qpkg service: phpMyAdmin is disabled.
    Unpack /mnt/HDA_ROOT/update_pkg/DSv3.tgz
    Starting UPnP MediaServer: (/share/MD0_DATA/.qpkg/TwonkyMedia/twonkystarter) AppleDouble,AppleDB,AppleDesktop,TemporaryItems,.@__thumb,@Recycle

    /etc/rcS.d/QS104tvmobili: line 7: /opt/var/lib/dbus/machine-id: No such file or directory
    Starting
    .
    [admin@Holly ~]#
    [admin@Holly ~]#
    [admin@Holly ~]#
    [admin@Holly ~]#
    [admin@Holly ~]#
    [admin@Holly ~]#
    [admin@Holly ~]#
    [admin@Holly ~]# cat /etc/config/raidtab
    raiddev /dev/md0
    raid-level 1
    nr-raid-disks 2
    nr-spare-disks 0
    chunk-size 4
    persistent-superblock 1
    device /dev/sda3
    raid-disk 0
    device /dev/sdb3
    raid-disk 1
    [admin@Holly ~]#
    [admin@Holly ~]#
    [admin@Holly ~]# cat /proc/mdstat
    Personalities : [raid1] [linear] [raid0] [raid6] [raid5] [raid4]
    md0 : active raid1 sda3[0]
    1463569600 blocks [2/1] [U_]
    bitmap: 147/175 pages [588KB], 4096KB chunk

    md2 : active raid1 sdb2[2] sda2[0]
    530128 blocks super 1.0 [2/2] [UU]

    md13 : active raid1 sda4[0] sdb4[1]
    458880 blocks [2/2] [UU]
    bitmap: 1/57 pages [4KB], 4KB chunk

    md9 : active raid1 sda1[0] sdb1[1]
    530048 blocks [2/2] [UU]
    bitmap: 1/65 pages [4KB], 4KB chunk

    unused devices:
    [admin@Holly ~]#
    [admin@Holly ~]# reboot
    [admin@Holly ~]#
    [admin@Holly ~]#
    [admin@Holly ~]#

  14. Hi!

    Thanks for writing this webpage. It has been very, very detailed and helped me a couple of times.

    We have RAID 1 in two older 2TB drives for the TS-212. However, one drive kept dropping off RAID, so we followed this page and rebuilt the drives sucessfully, and be online for some time (1 day, 1 week, 1 month). Then it will drop out again and run at degraded mode.

    Here is what happened this week

    Type Date Time Users Source IP Computer name Content
    2016/02/08 11:45:15 System 127.0.0.1 localhost [Mirror Disk Volume: Drive 1] RAID device in degraded mode.
    2016/02/08 08:59:37 Mirroring Disk Volume Drive 1 is completed
    2016/02/07 23:42:36 System 127.0.0.1 localhost [Mirror Disk Volume: Drive 1] RAID device in degraded mode.

    This is already the third time we rebuilt it. Has it happened to you or anyone? Thanks in advance for your help.

    Regards,
    Ray

    • Hi Ray,
      I read online that may be caused by several factor.
      If the disk goes to sleep and NAS need to wake it up, if the disk is not fast enough to report back to the controller, it may be marked as bad.
      you should check the SMART count of you HDD and see if you have a high number of recovery count: https://en.wikipedia.org/wiki/S.M.A.R.T.
      Also, in case the disk has some damaged sector, the physical controller try to retry the operation and this cause the NAS controller to mark the disk bad.
      If you want to dig into this topic, look for the difference between RED and GREEN disk on western digital website.
      RED series has Intelligent Error Recovery which try to recovery the bad sector without affecting the response time of the NAS controller.

      If it’s keep going out of sync, probably there may be something wrong with physical disk.

      You may try to download some utility from the manufacture website and perform some test before rebuilding the RAID.

      Thanks,
      Ing. Lele

      • Thank you so much for your prompt reply. This has been very helpful.
        These are old desktop HDDs, they are not NAS drives. I did use their utilities to scan the drives offline and did not find any issues.
        I contacted QNAP by email, and went through a lot of checks back-and-forth. They decided it is a hardware issue in their back panel, and asked me to RMA the NAS. I’m not quite sure on this analysis though. I will wait until it happened again, and replace the drive again – I have another 2TB handy.
        Again, thanks for your reply. You’ve been very helpful. Keep this webpage alive – it will help more people.

      • Now I realize why this takes place. Somehow this NAS was not on UPS. So every time there is a power blip at night, a disc dropped off mirroring. There was another NAS of the same model on the same power cord, and it always recovered well. QNAP offered return and replacement of this product, commenting a chip malfunctioned.

  15. great post.
    my situation: no raid1 recovery after disk replacement (disk1)
    solution: remove partition from disk1 & reboot

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s