FTP not showing all files conundrum!
In short when querying a FTP server for a list of files, a number of files were not being shown, even though we knew they were there!
A client contacted us and asked us why we were not processing all the files they had made available on their FTP server. All files we have processed were marked with DONE in the filename. I took a look but there were no files not marked with DONE. The client insisted there were, and sent us a screenshot of said files. Oddly enough we couldn’t see these files.
After playing around with permissions, active vs passive mode, SFTP etc we weren’t a step closer to seeing the files. Trying FileZilla, Explorer, and the command line FTP didn’t show the files. BUT when I typed ‘ls’ to list files, I DID see them. Typing the command ‘ls -l’ to see attributes caused the files NOT to be shown!
Outputting the file list from ‘ls -l’ gave me and output of 817 lines (files) which totalled 70080 bytes for the listing. ‘ls’ gave me 930 lines in a total of 41198 bytes. The smaller output is because ‘ls’ doesn’t output extended attributes for each file. Now I read on some forums that FTP servers can limit the number of files returned to 2000, but obviosuly we weren’t even near this total…so that wasn’t the problem. For some reason 113 files just weren’t showing.
We contacted a collegue in Cape Town (we’re in Johannesburg). Using EXACTLY the same settings and version of FileZilla he COULD see all the files, even the ones we couldn’t when using FileZilla. So next I tried connecting to a server outside our ISA proxy server, and connecting to the FTP server using the commandline FTP. Hey Presto! I could see all files. So I thought it was a proxy issue..but after some further investigation this didn’t make sense.
After a LOT of wasted time, I tried moving the already processed DONE files into an archive folder, thereby decreasing the number of files in the active folder. Then suddenly the previously ‘invisible’ files were shown. So something with the number of files or size of the ouput from the directory listing is causing problems.
So while we solved the problem, I still don’t know the cause of this odd FTP behaviour 😦