Node.js Buffer Module

Java FileNode.js Built-in Modules

Buffer is a class in Node.js used for handling binary data streams. In computers, all data is ultimately stored and transmitted in binary form, and Buffer provides a way to efficiently process these raw binary data in Node.js.

You can think of Buffer as a fixed-size "container" specifically designed to hold raw binary data. Unlike ordinary arrays in JavaScript, Buffer allocates a fixed-size block of memory, which makes it more efficient when handling large amounts of binary data.

Why do we need Buffer?

In web development, we often need to handle various binary data, such as:

  • File read and write operations
  • Data transmission in network communication
  • Image processing
  • Encryption and decryption operations

JavaScript was originally designed mainly for DOM operations in browsers, with limited ability to handle binary data. Node.js, as a server-side runtime, needs more powerful binary data processing capabilities, which is why the Buffer module exists.

Creating a Buffer

There are several ways to create a Buffer in Node.js:

1. Using Buffer.alloc()

Example

// Create a Buffer with a length of 10 bytes, filled with 0
const buf1 = Buffer.alloc(10);
console.log(buf1); // <Buffer 00 00 00 00 00 00 00 00 00 00>

2. Using Buffer.allocUnsafe()

Example

// Create a Buffer with a length of 10 bytes, but do not initialize the content
// May contain old data, so it is faster but less safe
const buf2 = Buffer.allocUnsafe(10);
console.log(buf2); // <Buffer content is uncertain>

3. Creating from an Array

Example

// Create a Buffer from an array
const buf3 = Buffer.from([0x62, 0x75, 0x66, 0x66, 0x65, 0x72]);
console.log(buf3); // <Buffer 62 75 66 66 65 72>

4. Creating from a String

Example

// Create a Buffer from a string
const buf4 = Buffer.from('Hello Node.js');
console.log(buf4); // <Buffer 48 65 6c 6c 6f 20 4e 6f 64 65 2e 6a 73>

Common Buffer Methods

Writing Data

Example

const buf = Buffer.alloc(5);

// Write a string
buf.write('Hello');
console.log(buf); // <Buffer 48 65 6c 6c 6f>

// Specify encoding when writing
const buf2 = Buffer.alloc(10);
buf2.write('Hello', 'utf8');
console.log(buf2); // <Buffer e4 bd a0 e5 a5 bd 00 00 00 00>

Reading Data

Example

const buf = Buffer.from('Hello World');

// Read as a string
console.log(buf.toString()); // Hello World

// Read part of the content
console.log(buf.toString('utf8', 0, 5)); // Hello

Copying a Buffer

Example

const buf1 = Buffer.from('Hello');
const buf2 = Buffer.from('World');
const buf3 = Buffer.alloc(10);

buf1.copy(buf3, 0); // Copy buf1 to the beginning of buf3
buf2.copy(buf3, 5); // Copy buf2 to the position starting at the 5th byte of buf3

console.log(buf3.toString()); // HelloWorld

Concatenating Buffers

Example

const buf1 = Buffer.from('Hello');
const buf2 = Buffer.from(' ');
const buf3 = Buffer.from('World');

const result = Buffer.concat([buf1, buf2, buf3]);
console.log(result.toString()); // Hello World

Buffer and Strings

Buffer and strings can be converted between each other, but attention must be paid to encoding. Common encodings are:

  • 'utf8' (default)
  • 'ascii'
  • 'base64'
  • 'hex'

Example

Example

// Convert string to Buffer
const str = 'Hello Buffer';
const buf = Buffer.from(str);

// Convert Buffer to string
console.log(buf.toString()); // Hello Buffer

// Use different encodings
console.log(buf.toString('hex')); // 48656c6c6f20427566666572
console.log(buf.toString('base64')); // SGVsbG8gQnVmZmVy

Practical Applications of Buffer

1. File Operations

Example

const fs = require('fs');

// Read file into Buffer
fs.readFile('example.txt', (err, data) => {
  if (err) throw err;
  console.log(data); // <Buffer ...>
  console.log(data.toString()); // File content
});

// Write Buffer to file
const content = Buffer.from('Hello File System');
fs.writeFile('output.txt', content, (err) => {
  if (err) throw err;
  console.log('File saved');
});

2. Network Communication

Example

const http = require('http');

http.createServer((req, res) => {
  const chunks = [];
 
  req.on('data', (chunk) => {
    chunks.push(chunk); // Collect data chunks
  });
 
  req.on('end', () => {
    const body = Buffer.concat(chunks); // Combine all data chunks
    console.log(`Received data: ${body.toString()}`);
    res.end('Data received');
  });
}).listen(3000);

3. Image Processing

Example

const fs = require('fs');
const sharp = require('sharp');

// Read image into Buffer
fs.readFile('input.jpg', (err, data) => {
  if (err) throw err;
 
  // Use sharp to process image data in Buffer
  sharp(data)
    .resize(200, 200)
    .toBuffer()
    .then(outputBuffer => {
      // Write the processed Buffer to a file
      fs.writeFile('output.jpg', outputBuffer, err => {
        if (err) throw err;
        console.log('Image processing complete');
      });
    });
});

Notes

  1. Memory management: Buffer operates directly on memory; allocating large Buffers can affect application performance.

  2. Security：Buffer.allocUnsafe()May contain sensitive data, so be careful when using it.

  3. Consistent encoding: When converting between Buffer and string, make sure to use the same encoding.

  4. Version differences: The Buffer API in older Node.js versions (v5.x and earlier) differs from the newer versions; it is recommended to use the latest API.

  5. Performance considerations: For large-scale data processing, proper use of Buffer can significantly improve performance.

Buffer is a core module in Node.js for handling binary data. Understanding its working principles and usage methods is crucial for developing efficient and reliable Node.js applications.

BufferModule Methods and Properties List

BufferIt is a core module in Node.js used for handling binary data. The following are the main methods and properties of the Buffer module:

Buffer Class Properties

Property/Method Description Version Introduced
Buffer.poolSize Pre-allocated internal Buffer instance size (bytes), modifiable 0.11.3
Buffer.from() Create a new Buffer from a string, array, or buffer 5.10.0
Buffer.alloc() Create an initialized Buffer of specified size 5.10.0
Buffer.allocUnsafe() Create an uninitialized Buffer of specified size (faster but unsafe) 5.10.0
Buffer.allocUnsafeSlow() Create an uninitialized Buffer (without using the shared memory pool) 5.12.0
Buffer.isBuffer() Check whether an object is a Buffer 0.1.101
Buffer.isEncoding() Check whether an encoding is valid 0.9.1
Buffer.concat() Combine an array of Buffers 0.7.11
Buffer.byteLength() Return the byte length of a string 0.1.90
Buffer.compare() Compare two Buffers 0.11.13

Buffer Instance Methods

Method Description Version Introduced
buf.length Byte length of the Buffer 0.1.90
buf.write() Write a string to the Buffer 0.1.90
buf.toString() Decode the Buffer to a string 0.1.90
buf.toJSON() Return the JSON representation of the Buffer 0.9.2
buf.equals() Compare whether two Buffers are equal 0.11.13
buf.compare() Compare Buffers and return the sort order 0.11.13
buf.copy() Copy Buffer data 0.1.90
buf.slice() Create a partial view of the Buffer 0.3.0
buf.includes() Check if the Buffer contains a value 5.3.0
buf.indexOf() Return the first occurrence position of a value 1.5.0
buf.lastIndexOf() Return the last occurrence position of a value 6.0.0
buf.fill() Fill the Buffer with a specified value 0.5.0
buf.readUIntLE() Read an unsigned little-endian integer 0.5.5
buf.readUIntBE() Read an unsigned big-endian integer 0.5.5
buf.readIntLE() Read a signed little-endian integer 0.11.15
buf.readIntBE() Read a signed big-endian integer 0.11.15
buf.readDoubleBE() Read a big-endian 64-bit double-precision floating-point number 0.11.15
buf.readDoubleLE() Read a little-endian 64-bit double-precision floating-point number 0.11.15
buf.readFloatBE() Read a big-endian 32-bit floating-point number 0.11.15
buf.readFloatLE() Read a little-endian 32-bit floating-point number 0.11.15
buf.writeUIntLE() Write an unsigned little-endian integer 0.5.5
buf.writeUIntBE() Write an unsigned big-endian integer 0.5.5
buf.writeIntLE() Write a signed little-endian integer 0.11.15
buf.writeIntBE() Write signed big-endian integer 0.11.15
buf.writeDoubleBE() Write big-endian 64-bit double-precision floating-point number 0.11.15
buf.writeDoubleLE() Write little-endian 64-bit double-precision floating-point number 0.11.15
buf.writeFloatBE() Write big-endian 32-bit floating-point number 0.11.15
buf.writeFloatLE() Write little-endian 32-bit floating-point number 0.11.15
buf.swap16() Swap 16-bit byte order 5.10.0
buf.swap32() Swap 32-bit byte order 5.10.0
buf.swap64() Swap 64-bit byte order 6.3.0
buf.keys() Create an iterator of Buffer keys 1.1.0
buf.values() Create an iterator of Buffer values 1.1.0
buf.entries() Create an iterator for [index, byte] 1.1.0

Encoding Types

Encoding types supported by Buffer:

Encoding Description
utf8 Multi-byte Unicode characters
utf16le 2 or 4-byte Unicode characters
latin1 ISO-8859-1
ascii 7-bit ASCII data only
base64 Base64 encoding
base64url URL-safe Base64 encoding
hex Hexadecimal encoding
ucs2 Alias for utf16le
binary Alias for latin1 (deprecated)

Example

// Create a Buffer
const buf1 = Buffer.alloc(10); // Initialize a zero-filled Buffer
const buf2 = Buffer.from('hello'); // Create from a string
const buf3 = Buffer.from([1, 2, 3]); // Create from an array

// Write and read
buf1.write('Node.js');
console.log(buf1.toString('utf8', 0, 6)); // 'Node.js'

// Compare Buffers
const bufA = Buffer.from('A');
const bufB = Buffer.from('B');
console.log(bufA.compare(bufB)); // -1

// Concatenate Buffers
const concatBuf = Buffer.concat([bufA, bufB]);

Java FileNode.js built-in modules

Other extensions