Upload CSV File from S3 bucket to SFTP Server in NodeJS

Posted By : Ankit Uniyal | 26-Dec-2017

nodejs

In this blog, we will cover up how to upload CSV file from S3 bucket to SFTP Server using NodeJS. But before that let's have a quick look on how we can set up S3 bucket and it's configurations.

Steps to follow for creating S3 bucket :

1.Sign In on AWS S3 console, below is the url :

https://console.aws.amazon.com/s3/

2.Then Click on Create Bucket.

3.Now, on Name and Region field, type your bucket name and make sure the bucket name should be unique which never used for any other bucket name and then select your AWS Region.

4.Then Choose Next and then Next and after that on Review Page click on Create bucket.

5.You can also grant public access right to the bucket but that generally should not to follow.

If we want to provide the S3 bucket API access right to any lambda function, then we can add Policy to that lambda from IAM user AWS console and we need to add policy for every s3 actions or any particular S3 actions.

Now, below are the two steps which we need to follow to upload CSV file from S3 bucket to SFTP server :

1.Get S3 bucket CSV file content :

var getS3UploadedCSVFileContent = function(next, results) {
    var filePath = 'your_file_path';
    var s3 = new AWS.S3();
    var params = {
        Bucket: 'your_bucket_name',
        Key: filePath
    }
    s3.getObject(params, function(err, data) {
        if (err) {
            console.log("Error in getting CSV file from S3 bucket", err);
        } else {
            console.log("Content is", data);
            var dataObject = data.Body.toString();
            next(null, dataObject);
        }
    })
}

2.Upload CSV file on SFTP Server :

I have used 'ssh2' npm module to upload CSV file on SFTP server.

var uploadCSVFileOnSftpServer = function(next, s3FileStreamContent) {
    var filePath = 'your_file_path';
    var Client = require('ssh2').Client;
    var connSettings = {
        host: 'your_server_host_name',
        port: 22,
        username: 'user_name',
        password: 'password'
    };
    var conn = new Client();
    conn.on('ready', function() {
        conn.sftp(function(err, sftp) {
            if (err) {
                console.log("Errror in connection", err);
            } else {
                console.log("Connection established", sftp);
                var options = Object.assign({}, {
                    encoding: 'utf-8'
                }, true);
                var stream = sftp.createWriteStream(filePath, options);
                var data = stream.end(s3FileStreamContent);
                stream.on('close', function() {
                    console.log("- file transferred succesfully");
                    conn.end();
                    next(null, true);
                });
            }
        });
    }).connect(connSettings);
}

Note: We should always make sure that we close the sftp connection after the process is complete.

Thanks