#java io

In the previous part of our Java I/O series, we introduced the concept of streams and explored the foundational ideas that drive Java’s input and output operations. We discussed how Java abstracts the complexities of data handling across various sources,

Introduction In the previous part of our Java I/O series, we introduced the concept of streams and explored the foundational ideas that drive Java’s input and output operations. We discussed how Java abstracts the complexities of data handling across various sources, such as files and network connections, into a unified stream-based model. In this article, we will dive deep into one of the core…

♦ 자바 I/O 의 개요 ◊ I/O 용어 정리  자바 I/O 의 기준은 항상 자바 애플리케이션이다.   소스  ⇒  자바 애플리케이션  ⇒  목적지

소스(source) 자바 애플리케이션에서 데이터를 읽어 들이는 대상을 의미한다. 키보드, 파일, 네트워크 등이 있다. 

목적지(destination) 자바 애플리케이션에서의 데이터 출력 대상을 의미한다. 모니터, 파일, 네트워크 등이 있다. 

입력(input) 소스에서 자바 애플리케이션으로 데이터를 읽어 들이는 작업을 의미한다. 입력 스트림(input stream) 이라고 한다. 

출력(output) 자바 애플리케이션에서 목적지로데이터를 출력시키는 작업을 의미한다. 출력 스트림(output stream) 이라고 한다. 

  ◊ 자바의 I/O 데이터 종류 byte 자바…

New Post has been published on http://heapload.com/computer-science/io-in-java

IO in Java

IO in Java

Java has very optimized and efficient packages for IO (Input and Output) operations. Java uses concept of stream (packages of data/information) for input and output tasks that are effective and supports many data types.

For reading data/information, Java invokes InputStream, and for writing, it uses OutputStream objects.

Code Examples:

Byte Streams

Java byte stream is one the most common approaches for Input and Output operations. This method converts information in the form of 8-bit bytes packages and then transfer them to the destination.

package java.examples; import java.io.FileInputStream; import java.io.FileOutputStream; import java.io.IOException; /** * * @author heapload.com */ public class byteStreamExample public static void main(String args[]) throws IOException FileInputStream input = null; FileOutputStream output = null; try input = new FileInputStream("a.txt"); output = new FileOutputStream("b.txt"); int counter; while ((counter = input.read()) != -1) output.write(counter); finally if (input != null) input.close(); if (output != null) output.close();

java-byte-stream

Procedure:

Create a new class and name it as byteStreamExample.java

Make two objects for FileInputStream and FileOutputStream with file name.

Iterate the FileInputStream object until it returns -1 as end of file.

Write all the stream information into FileOutputStream object.

Finally, close both the streaming objects.

Character Streams

Java character stream supports 16-bit (twice than byte streams) Unicode streams. This method converts information in the form of 16-bit Unicode packages and then transfer them to the destination. Like java byte streams, it has FileReader and FileWriter classes for IO purpose.

java-byte-stream-and-character-stream-comparision

import java.io.FileReader; import java.io.FileWriter; import java.io.IOException; /** * * @author heapload.com */ public class characterStreamExample public static void main(String args[]) throws IOException FileReader in = null; FileWriter out = null; try in = new FileReader("a.txt"); out = new FileWriter("b.txt"); int counter; while ((counter = in.read()) != -1) out.write(counter); // 16-bit unicode stream (twice than byte stream) finally if (in != null) in.close(); if (out != null) out.close();

java-character-stream

Procedure:

Create a new class and name it as characterStreamExample.java

Make two objects for FileReader and FileWriter with file name.

Iterate the FileReader object until it returns -1 as end of file.

Write all the stream information into FileWriter object.

Finally, close both the streaming objects.

Read Complete File Once

If the file is small, then you can also avoid iterating InputStream object with this simple approach.

import java.io.File; import java.io.FileInputStream; /** * * @author heapload.com */ public class readFileOnce public static void main(String args[]) try File file = new File("a.txt"); FileInputStream fis = new FileInputStream(file); byte[] data = new byte[(int) file.length()]; fis.read(data); fis.close(); String str = new String(data, "UTF-8"); catch (Exception ex) System.out.println("Error In File Reading" + ex.toString());

java-read-file-once

Procedure:

Create a new class and name it as readFileOnce.java

Make two objects for File and FileInputStream.

Make a byte array with the size of the source file.

Write all the stream information once into FileInputStream object.

IO in Java Java has very optimized and efficient packages for IO (Input and Output) operations. Java uses concept of stream (packages of data/information) for input and output tasks that are effective and supports many data types. For reading data/information, Java invokes InputStream, and for writing, it uses OutputStream objects. Code Examples: Byte Streams Java byte stream is one the most…

Java IO İle Dosya Silme İşlemi

Merhaba Arkadaşlar Bugün Java’da geçici dosya oluşturduğumuzda silmemiz gerekir. Bu silme işlemini yapmak için Java IO API’sini kullanmalıyız. Aslında oldukça basit bir işlem. Kodları okuyarak kesinlikle anlayacaksınız. Aynı şekilde sınıflardan yararlanarak dosyayı silme işlemini yapacağız. (more…)

Stream based IO model of Java

Before we take up NIO its important that we refresh our understanding of the stream based IO model. It forms the core to java.io - the fundamental package which "provides for system input and output through data streams and serialization."

Let us analyse the model with focus on data streams as this is where the need of an improved treatment of IO in Java was felt.

Data Stream is a sequence of data units that can be accessed in order. The unit of data is fundamentally either a byte or a UTF-8 encoded character. Now we can either read from a stream or write to it. Any application that requires to read or write data would ultimately function at the level of data streams. A typical program reads data from a "SOURCE" in the form of an "INPUT" stream and delivers it to the "DESTINATION" as an "OUTPUT" stream. This approach generally translates into a loop that reads the input stream and writes to the output stream byte by byte or character by character. The following diagram attempts to elucidate the same:

Implications of this "stream oriented" approach:

1.) We cannot move back and forth. Data stream is actually a continuous flow of data. Unlike an array it is not indexed and the data units (bytes or characters) are not cached anywhere. So we simply cannot traverse the stream at will. If while writing an output stream you suddenly have to rewrite a portion of it, it becomes virtually impossible. Similarly while reading an input stream if you wish to reread a portion of the former half, you are hopelessly stuck. All that you can do to circumvent this is to cache your data in a buffer and then move back and forth. 2.) The thread remains blocked. In java.io the streams are "blocking". That is, if a thread is performing a read or a write operation it shall remain blocked until there is some new data to read or it is finished with the writing of data. More so, that if two threads attempt to read from a single input stream ,one beginning its read() after another the second thread will be forced to begin from where the first one leaves. 3.) Impedance mismatch. As has already been discussed in the previous blog, "impedance mismatch" is linked to the byte-by-byte or character-by-character treatment of data streams by the JVMs. To put it succinctly, the OS carts data in large chunks. These chunks need to be broken down by java.io before they can be used. As a result there is a huge mismatch in the speed with which data can be delivered by the OS to java.io and the speed with which java.io actually processes it further. To quote,Ron Hitchens from Java NIO, "The operating system wants to deliver data by the truckload. The java.io classes want to process data by the shovelful." This indeed appropriately summarises the performance spoiler java.io can sometimes be. 4.) Every connection has its own thread. In the Stream based model each connection (i.e., your input sources or output destinations) will require a separate thread. It is a direct fallout of the blocking nature "java.io" streams. So to deal with multiple connections at a time you shall need multiple threads. One for each. It turns out be a cumbersome when the amount of data in consideration is small and the connections are in thousands. 5.) Overhead generated by each IO request. Well this is one of the factors that result in the aforementioned mismatch of impedance. While the program loops the data byte-by-byte (or character-by-character) each such read or write request often triggers expensive operations like disk access or network activity. Again if we need to minimize these overheads we need to buffer the data first.

With the vision to rectify the inefficiencies of the stream oriented approach an improved model was developed and packaged as Java NIO!

[References:

This blog post would explain how to use Akka actor as an input for Spout in any given Storm topology. Since Spouts are designed to tap out data from any given data source I thought of Akka Actor could also be an eligible candidate as supplier of data for a Spout in a Storm topology.

Data Staging The whole idea is depicted in the above diagram. FileReader reads a file line by line and post that toa TCP port. For each line posted the Actor holds it to the local queue.

Data Processing When the Storm cluster gets started the Spout in could read from this Actor based queue (one at a time) and process it with various combinations of Storm Bolt/s.

Why a Actor Based Queue for Storm? When there is a need to feed a big File as input to Storm's topology , Spout it wouldn't be a suitable abstraction, since a Storm toplogy essentially runs in multiple threads across multiple nodes (in a cluster environment) file reading behaviour if embedded in Spout we would read the same file multiple times which not we want and hence the we delegate that to a Actor which holds a queue of lines read by the FileReader.

Code for Akka Based Queue

case class ParsedLineData(str: String) class TCPServerActor(port: Int) extends Actor { val queue = new Queue[String] override def preStart { IOManager(context.system) listen new InetSocketAddress(port) } def receive = { case ParsedLineData(str) => queue.enqueue(str) case IO.NewClient(server) => server.accept() case IO.Read(rHandle, bytes) => rHandle.asSocket.write(ByteString(queue.dequeue() + "\n".getBytes()).compact) } }

Code that kickstarts Queue Actor

object TCPServerActorApp extends App { def startTCP() { println("exposing new TCP port for message queue @ 6789") val server = ActorSystem().actorOf(Props(new TCPServerActor(6789))) val fileSrc = scala.io.Source.fromFile(this.getClass().getResource("/gitsActorRead.csv").getPath()) fileSrc.getLines().foreach(line => { server.!(ParsedLineData(line)) }) } startTCP() }

Message Reading Spout

class LineBasedProcessingSpout() extends StormSpout(outputFields = List("Line")) { def nextTuple { val next = nextTupleImpl emit(next) } def nextTupleImpl = { val socket = new Socket("localhost", 6789) val br = new BufferedReader(new InputStreamReader(socket.getInputStream())) val dos = new DataOutputStream(socket.getOutputStream()) dos.writeChars("\n") val x = br.readLine() dos.close() br.close() x } }

Sometime before I was wondering how log analysis system works , I started to investigate from reading a log file. Assuming the size of the log files are very large and a single thread cannot afford to read the entire data at once hence ready only a chunk of data would be a better idea. 

Thats the moral of this blog post, when we say reading data in chunk we should keep remind ourselves the amount of data read / or a pointer the reader could remember so it continue reading the next  chunk on subsequent calls. Ok Ok I could catch your mind voice , enough talking here show me the code !!! , here we go .... 

As you can see Iam reading a URL and specify the blocksize

and the skipsize. The blocksize is fixed to fit in to the array

the skipsize helps to read the stream by ommiting skipsize * blocksize 

no of arrays for eg if the skipsize is 3 , the code omits 3 * 1024 bytes 

from the stream. The skipsize is supposed to be given by the caller

to as to slide through the bytes to omit. 

  But what I get by reading a file " chunk by chunk "

By reading data by chunk the file stream could be read indefinetly while dealing

with large file 

Read any chunk of a file either from beginning , this would lead to spawning

multiple thread / actors to read the file hence making faster reads.

  Conclusion

I hope I have given a bit of intro but I would like to give