How to efficiently read file line by line in Node.js

Photo by Mika Baumeister on Unsplash

This article focus on a few common ways to read a file in Node.js (also using typescript for the code examples).

1. readline native module
2. EventStream open source npm library

A common mistake when reading a file line-by-line in Node.js is to read the whole file into memory and then split its content by line breaks. Doing this might overload your system, especially when dealing with large files.

Here is a sample file with many lines which needs to be processed line by line.

# app.log
Line 1
Line 2
Line 3
Line 4
Line 5
...

readline

Node.js has a native module readline which provides an interface for reading data from stream one line at a time.

We could utilise this behaviour for reading from the file stream and process each line separately.

Basically, the steps to achieve this are

1. create the realine object using createInterface function. While doing this, we have to supply the file stream.
2. bind a listener to line event

The following snippet uses Node.js native readline module to read the above file into a stream and print line by line.

import { createReadStream } from 'fs'
import { createInterface } 'readline'
const readFile = () => {
  // Step 1
  const rlInterface = createInterface({
    input: createReadStream('app.log'),
    output: process.stdout,
    terminal: false // to indicate this is not TTY
  })
  // Step 2
  rlInterface.on('line', (line) => {
    console.log(line)
  })
}

Invoking readFile function produces the following output

Console
Line 1
Line 2
Line 3
Line 4
Line 5
...

This approach outperforms the usual readFileSync although Node.js still attempts to hold the entire file in memory when streaming the file input to output.

We would need a new solution if you have a really large file to process line by line.

EventStream

EventStream is a popular NPM library with a promise to make creating and working with streams easy.

To use the EventStream, install using npm

npm i --save-exact event-stream

Optionally, to install the types

npm i --save-exact --save-dev @types/event-stream

Once the EventStream is installed, We would just pipe the read stream to a listener to read a file line by line.

Below is a code snippet that usesevent-stream to acheive the same as the one in readline snippet above.

import { createReadStream } from 'fs'
import { split } from 'event-stream'
const readFileEventStream = () => {
  createReadStream('app.log')
    .pipe(split())
    .on('data', console.log)
}

Conclusion

Here I presented two different ways of reading a file line by line, one with a Node.js native module and another with a popular open source library.

Pick one of the approaches depending on your needs. And if the data size gets big, it is always wise to use the EventStream library not to crash the Node server.