I've been avoiding starting a project myself which will require SFTP. Passing text files around via SFTP is how a lot of the world works. It's still painful to implement this, and makes the transfer of data slow (polling for new files to process etc). This introduces a big delay in any data feedback cycles. Managing users with system accounts is painful too.
This project looks to alleviate a lot of these problems.
It'd be great to be able to register webhooks so that it could send events to external systems. Ideally I'd like to know when a file is created/deleted etc without having to walk directories on the sftp server on a regular basis.
Yes, if you can avoid polling by getting events it can be a good system. But really you want to be notified when a file is closed successfully, not closed after a timeout of other error.
For me, in a previous system I uploaded as a .tmp file, then the uploaded renames to .zip at the end of the upload. The rename is atomic so no chance of the consumer reading a partial file.
I was dealing in CSV, TSV and XML files. I chose to upload them all in a zip wrapper not only because they compress well, but also because each file has a CRC and the file won’t be extracted if the file is damaged or truncated. It’s hard to tell if a raw CSV file has been truncated.
just drop files with guids into stage dirs and send all staged files over to another stage dir on the other side. When successfullly archive on both ends of the transfer.
This project looks to alleviate a lot of these problems.
It'd be great to be able to register webhooks so that it could send events to external systems. Ideally I'd like to know when a file is created/deleted etc without having to walk directories on the sftp server on a regular basis.