December 13, 2024

Fedora 41 -- new disk for trona

Nothing specific here to Fedora 41, at least I don't think so.

I have two 4T drives in this machine. I mount them as /u1 and /u2 and /u2 is a mirror of /u1. I have been getting messages from the u1 drive like so: 

Dec  8 04:14:46 trona smartd[1009]: Device: /dev/sdc [SAT], 2 Currently unreadable (pending) sectors
Dec  8 04:14:46 trona smartd[1009]: Device: /dev/sdc [SAT], 2 Offline uncorrectable sectors
Dec  8 04:44:46 trona smartd[1009]: Device: /dev/sdc [SAT], 2 Currently unreadable (pending) sectors
Dec  8 04:44:46 trona smartd[1009]: Device: /dev/sdc [SAT], 2 Offline uncorrectable sectors
The disk has not actually failed, but the word in general is that when you start seeing things like this, doom is right around the corner.

So, I purchased a replacement drive. It is a WD4000FYYZ. Date is marked 20 May 2014 on the drive, so this is either old stock or a refurbished drive. I edit fstab so that it mounts "sdd" as /u1, swap the "sdc" drive and boot back up. My system is installed on a pair of SSD which are sda and sdb so this works.

Partition the new drive

The old (good) disk is partitioned like this: 
Disk /dev/sdd: 3.64 TiB, 4000787030016 bytes, 7814037168 sectors
Disk model: WDC WD4000FYYZ-0
Device        Start        End    Sectors  Size Type
/dev/sdd1      2048      98303      96256   47M Linux filesystem
/dev/sdd2     98304   31445312   31347009 14.9G Linux swap
/dev/sdd3  31445313 7814037134 7782591822  3.6T Linux filesystem
Good old fdisk is limited to 2T drives, so I need to use something else to partition the new disk, namely "parted" 
parted /dev/sdc
The trick is using the "mkpart" command. Everything is referenced to the start of the disk, which is pretty inconvenient. I have notes about a previous trial and error to sort this all out. I give a start somewhere inside the prior partition, and then parted scolds me and offers to set the start to immediately follow that partition, which is what I want. I ended up with pretty much the following. You can type "mklabel" at any time to start over.

A hot tip. I am writing this page as I do all of this, so I use vim to type the following, then cut and paste it into the window with parted which avoids stupid typos. I do this a lot for boring sysadmin tasks like this. 

print
mklabel GPT
mkpart "extra" ext4 1 48m
mkpart "swap" linux-swap 40m 16.1g
mkpart "u3" ext4 8g -1s
print
quit

Yes, you just quit and it is for real, unlike fdisk.
After this was done I did: 

mkfs -t ext4 /dev/sdc1
mkfs -t ext4 /dev/sdc3

Copy the data

We have a bit over 3T to copy: 
df /u1
Filesystem      1K-blocks       Used Available Use% Mounted on
/dev/sdd3      3829075716 2651532844 982961696  73% /u1
We will leave the new disk on /u2 for now and edit fstab to put it back to /u1 when the copy is all done. 
su
mount /dev/sdc3 /u2
cd /u1
umask 0
tar cf - . | ( cd /u2 ; tar xpf - )
For some insane reason, it is extracting all the files as owned by root and with restricted permissions: 
drwx------  10 root root  4096 Dec 13 09:57 android-ndk-r8d
drwx------   7 root root  4096 Dec 13 09:57 idea-IC-231.9161.38
drwx------   2 root root 16384 Dec 13 09:56 lost+found
drwx------  32 root root  4096 Dec 13 09:58 tom_archive
I use "tar cf /u2/junk.tar ." to see what it is putting in the tar file and all the ownerships and permissions in the file are correct. There are flags "-o" and "-m". I don't recall ever having to use them before.
We do this to redo the mess: 
umount /u2
mkfs -t ext4 /dev/sdc3
mount /dev/sdc3 /u2
cd /u1
tar cf - . | ( cd /u2 ; tar xpf - )
#tar cf - . | ( cd /u2 ; tar xpf - -o -m )
But it doesn't work. Everything still gets created owned by root and as if there is a umask of 077 (turning off all access for other users). This smells like some ill conceived security "feature" that the Fedora group has sneaked in.

Time for some experiments

I hate it when things don't "just work". I try this: 
cd /u1/home
tar cpf /u2/tom.tar ./tom
cd /u2/home
tar xpvf ../tom.tar
And this works! All the files in /u2/home/tom have proper ownerships and permissions. Why would tar extraction from a file work, but not from a pipeline?

I try a reboot with /u2 in the fstab -- no better.
I try this: 

root@trona:/u1# tar cf - . | tar tvf - | more
drwxr-xr-x tom/tom           0 2024-08-28 17:34 ./
drwxr-xr-x tom/tom           0 1970-01-20 05:40 ./idea-IC-231.9161.38/
drwxr-xr-x tom/tom           0 1970-01-20 05:40 ./idea-IC-231.9161.38/lib/
This confirms that the tar create is working as it should, placing both ownerships and permissions into the "file".

What about selinux, my mortal enemy?

I look at /etc/selinux/config and see "SELINUX=disabled", but a comment catches my eye that tells me there is now a bit more required, you need to pass "selinux=0" to the kernel to entirely get rid of it. They suggest using the following command: 
grubby --update-kernel ALL --args selinux=0
And of course we will need to reboot again. This does not do it.

Other ideas and various experiments

What about flipping the tar subshell thing around: 

cd /u2
( cd /u1 ; tar cf - . ) | tar xpvf -
Nope -- this also fails.

Now try this: 

cd /home/tom
( cd /u1 ; tar cf - . ) | ( cd /u2 ; tar xpf - )
Nope -- no magic in this.
So, let's try this: 
rsync -auv --exclude-from=/u1/Backup/trona_excludes --delete /u1/ /u2
No! This has the same trouble.

I considered letting this run and writing a script to fix ownership and permissions for the top level directories, but then discovered that there were also issues deeper into the filesystem.

I ended up doing this: 

mount /u2
chown tom:tom /u2
cd /u1
cp -arp . /u2
It took over 10 hours, but the result seems fine.

Final analysis

I have a theory, so far unproven.

The theory is that tar, when finished, would have made it all right. I was looking at things while tar was still busy, for example looking at a huge tar file, as well as directories that files were still being written into. I would have had to let it run for 10 hours and then taken a look.

This would also explain why my small scale experiments all worked. I looked at permissions once tar has finished.

At this point I don't really care and have moved on to other things.

Have any comments? Questions? Drop me a line!

Adventures in Computing / tom@mmto.org