Generating maven artifacts from Kafka sourcecode

All commands to be performed on the root directory of the kafka source.
e.g. /home/user/kafka-0.7.2-incubating-src

Follow all the steps::

• Retrieve kafka source code (0.7.2 preferred)

          svn co http://svn.apache.org/repos/asf/incubator/kafka/trunk kafka
          or
          https://www.apache.org/dyn/closer.cgi/incubator/kafka/kafka-0.7.2-incubating/kafka-0.7.2- incubating-src.tgz

• Perform initial update for Maven. (Run at root of kafka source folder) 

          Command: ./sbt update

• Update project/build/KafkaProject.scala. Copy the bold code and place at the location specified in red line. Dont copy the red lines

<KafkaProject : CODE : Top of the KafkaProject class, after the first blank line>

          lazy val multimoduleProject =
          <artifactId>{name}</artifactId> ++
          <packaging>pom</packaging> ++
          <modules>{
          subProjects.keys.map(module => <module>{module}</module>)
          }</modules>

          object PackagingTypeChanger extends RuleTransformer(new RewriteRule() {
          override def transform(node: Node): Seq[Node] = node match {
          case elem @ Elem(_, "packaging", _, _, _) => multimoduleProject
          case other => other
          }
          })

          override def pomPostProcess(pom: Node): Node =
          if(subProjects.size > 0) PackagingTypeChanger(pom)
          else pom

          </KafkaProject : CODE>

          <CoreKafkaProject : CODE : Top of the CoreKafkaProject class, after the first blank line>

          override def pomExtra =

          <build>

            <plugins>

              <plugin>

                <groupId>org.scala-tools</groupId> 

                <artifactId>maven-scala-plugin</artifactId>

                <version>2.15.0</version>

                <executions>

                  <execution>

                    <id>scala-compile-first</id>

                    <phase>process-resources</phase>

                    <goals>

                      <goal>add-source</goal>

                      <goal>compile</goal>

                    </goals>

                  </execution>

                  <execution>

                    <id>scala-test-compile</id>

                     <phase>process-test-resources</phase>

                     <goals>

                       <goal>testCompile</goal>

                     </goals>

                 </execution>

              </executions>

           </plugin>

           <plugin>

             <artifactId>maven-compiler-plugin</artifactId>

             <version>2.3.2</version>

             <configuration>

               <source>1.6</source>

               <target>1.6</target>

             </configuration>

           </plugin>

       </plugins>

    </build>

    </CoreKafkaProject : CODE>

    <KafkaPerfProject : CODE : Top of the KafkaPerfProject class, after the first blank line>

    override def pomExtra =

    <build>

      <plugins>

       <plugin>

       <groupId>org.scala-tools</groupId>

       <artifactId>maven-scala-plugin</artifactId>

       <executions>

         <execution>

             <goals>

               <goal>compile</goal>

               <goal>testCompile</goal>

             </goals>

           </execution>

         </executions>

        <version>2.9.1</version>

      </plugin>

    </plugins>

   </build>

   </KafkaPerfProject : CODE>

    <ContribProject : CODE : Top of the ContribProject class, after the first blank line>

    lazy val multimoduleProject =

    <artifactId>{name}</artifactId> ++

    <packaging>pom</packaging> ++

    <modules>{

    subProjects.keys.map(module => <module>{module}</module>)

    }</modules>

    object PackagingTypeChanger extends RuleTransformer(new RewriteRule() {

    override def transform(node: Node): Seq[Node] = node match {

    case elem @ Elem(_, "packaging", _, _, _) => multimoduleProject

    case other => other

    }

    })

    override def pomPostProcess(pom: Node): Node =

    if(subProjects.size > 0) PackagingTypeChanger(pom)

    else pom

    

    </ContribProject : CODE>

• Build the poms (run at root of kafka source folder)

Command : ./sbt make-pom

• Copy the generated poms to the respective root folders

          Commands :

          cp ./target/scala_2.8.0/kafka_2.8.0-0.7.2.pom pom.xml
          cp ./core/target/scala_2.8.0/kafka-0.7.2.pom core/pom.xml
          cp ./contrib/target/scala_2.8.0/contrib_2.8.0-0.7.2.pom contrib/pom.xml
          cp ./contrib/hadoop-producer/target/scala_2.8.0/hadoop-producer_2.8.0-0.7.2.pom contrib/hadoop-producer/pom.xml
          cp ./contrib/hadoop-consumer/target/scala_2.8.0/hadoop-consumer_2.8.0-0.7.2.pom contrib/hadoop-consumer/pom.xml
          cp ./perf/target/scala_2.8.0/kafka-perf-0.7.2.pom perf/pom.xml
          cp ./examples/target/scala_2.8.0/kafka-java-examples-0.7.2.pom examples/pom.xml

• Modify ./pom.xml as follows

Modify (<groupId>) : kafka -> org.apache.kafka

Remove :

<dependency>

<groupId>kafka</groupId>

<artifactId>core-kafka_2.8.0</artifactId>

<version>0.7.2</version>

<scope>compile</scope>

</dependency>

<dependency>

<groupId>kafka</groupId>

<artifactId>contrib_2.8.0</artifactId>

<version>0.7.2</version>

<scope>compile</scope>

</dependency>

<dependency>

<groupId>kafka</groupId>

<artifactId>perf_2.8.0</artifactId>

<version>0.7.2</version>

<scope>compile</scope>

</dependency>

<dependency>

<groupId>kafka</groupId>

<artifactId>java-examples_2.8.0</artifactId>

<version>0.7.2</version>

<scope>compile</scope>

</dependency>

• Modify ./core/pom.xml as follows

Modify (<groupId>) : kafka -> org.apache.kafka

Modify (<artifactId>) : core-kafka_2.8.0-> core

Add (under <project> tag):

<parent>

<groupId>org.apache.kafka</groupId>

<artifactId>kafka</artifactId>

<version>0.7.2</version>

</parent>

<properties>

<maven.compiler.source>1.6</maven.compiler.source>

<maven.compiler.target>1.6</maven.compiler.target>

<encoding>UTF-8</encoding>

<scala.version>2.9.1</scala.version>

</properties>

Add (under <dependencies> tag)

<dependency>

<groupId>org.scala-lang</groupId>

<artifactId>scala-library</artifactId>

<version>${scala.version}</version>

</dependency>

<dependency>

<groupId>org.easymock</groupId>

<artifactId>easymock</artifactId>

<version>2.5</version>

</dependency>

<!-- Test -->

<dependency>

<groupId>junit</groupId>

<artifactId>junit</artifactId>

<version>4.8.1</version>

<scope>test</scope>

</dependency>

<dependency>

<groupId>org.scala-tools.testing</groupId>

<artifactId>specs_${scala.version}</artifactId>

<version>1.6.9</version>

<scope>test</scope>

</dependency>

<dependency>

<groupId>org.scalatest</groupId>

<artifactId>scalatest_${scala.version}</artifactId>

<version>1.6.1</version>

<scope>test</scope>

</dependency>

Modify (<dependency>/<log4j>/version) : 1.2.15 -> 1.2.16

• Modify ./perf/pom.xml as follows

Modify (<groupId>) : kafka -> org.apache.kafka

Modify (<artifactId>) : perf_2.8.0-> perf

Add (under <project> tag):

<parent>

<groupId>org.apache.kafka</groupId>

<artifactId>kafka</artifactId>

<version>0.7.2</version>

</parent>

<properties>

<maven.compiler.source>1.6</maven.compiler.source>

<maven.compiler.target>1.6</maven.compiler.target>

<encoding>UTF-8</encoding>

<scala.version>2.9.1</scala.version>

</properties>

Add (under <dependencies> tag)

<dependency>

<groupId>org.scala-lang</groupId>

<artifactId>scala-library</artifactId>

<version>${scala.version}</version>

</dependency>

<!-- Test -->

<dependency>

<groupId>junit</groupId>

<artifactId>junit</artifactId>

<version>4.8.1</version>

<scope>test</scope>

</dependency>

<dependency>

<groupId>org.scala-tools.testing</groupId>

<artifactId>specs_${scala.version}</artifactId>

<version>1.6.9</version>

<scope>test</scope>

</dependency>

<dependency>

<groupId>org.scalatest</groupId>

<artifactId>scalatest_${scala.version}</artifactId>

<version>1.6.1</version>

<scope>test</scope>

</dependency>

Modify (<dependency>/<core-kafka_2.8.0>/groupid) : kafka -> org.apache.kafka

Modify (<dependency>/<core-kafka_2.8.0>/artifactid) : core-kafka_2.8.0 -> core

Modify (<dependency>/<log4j>/version) : 1.2.15 -> 1.2.16

• Modify ./examples/pom.xml as follows

Modify (<groupId>) : kafka -> org.apache.kafka

Modify (<artifactId>) : java-examples_2.8.0 -> examples

Add (under <project> tag):

<parent>

<groupId>org.apache.kafka</groupId>

<artifactId>kafka</artifactId>

<version>0.7.2</version>

</parent>

<properties>

<maven.compiler.source>1.6</maven.compiler.source>

<maven.compiler.target>1.6</maven.compiler.target>

<encoding>UTF-8</encoding>

<scala.version>2.9.1</scala.version>

</properties>

Remove (under <dependencies> tag) (this is a duplicate entry)

<dependency>

<groupId>kafka</groupId>

<artifactId>core-kafka_2.8.0</artifactId>

<version>0.7.2</version>

<scope>compile</scope>

</dependency>

Modify (<dependency>/<log4j>/version) : 1.2.15 -> 1.2.16

Modify (<dependency>/<core-kafka_2.8.0>/groupid) : kafka -> org.apache.kafka

Modify (<dependency>/<core-kafka_2.8.0>/artifactid) : core-kafka_2.8.0 -> core

• Modify ./contrib/pom.xml as follows

Modify (<groupId>) : kafka -> org.apache.kafka

Add (under <project> tag):

<parent>

<groupId>org.apache.kafka</groupId>

<artifactId>kafka</artifactId>

<version>0.7.2</version>

</parent>

Remove :

<dependency>

<groupId>kafka</groupId>

<artifactId>hadoop-consumer_2.8.0</artifactId>

<version>0.7.2</version>

</dependency>

<dependency>

<groupId>kafka</groupId>

<artifactId>hadoop-producer_2.8.0</artifactId>

<version>0.7.2</version>

</dependency>

• Modify ./contrib/hadoop-consumer/pom.xmlas follows

Modify (<groupid>) : kafka -> org.apache.kafka.contrib

Modify (<artifactId>) : hadoop-consumer_2.8.0 -> hadoop-consumer

Add (under <project> tag):

<parent>

<groupId>org.apache.kafka</groupId>

<artifactId>contrib</artifactId>

<version>0.7.2</version>

</parent>

<properties>

<maven.compiler.source>1.6</maven.compiler.source>

<maven.compiler.target>1.6</maven.compiler.target>

<encoding>UTF-8</encoding>

<scala.version>2.9.1</scala.version>

</properties>

Modify (<dependency>/<log4j>/version) : 1.2.15 -> 1.2.16

Modify (<dependency>/<core-kafka_2.8.0>/groupid) : kafka -> org.apache.kafka

Modify (<dependency>/<core-kafka_2.8.0>/artifactid) : core-kafka_2.8.0 -> core

• Modify ./contrib/hadoop-producer/pom.xml as follows

Modify (<groupid>) : kafka -> org.apache.kafka.contrib

Modify (<artifactId>) : hadoop-producer_2.8.0 -> hadoop-producer

Add (under <project> tag):

<parent>

<groupId>org.apache.kafka</groupId>

<artifactId>contrib</artifactId>

<version>0.7.2</version>

</parent>

<properties>

<maven.compiler.source>1.6</maven.compiler.source>

<maven.compiler.target>1.6</maven.compiler.target>

<encoding>UTF-8</encoding>

<scala.version>2.9.1</scala.version>

</properties>

Remove (under <dependencies> tag) (we need version 0.9.2)

<dependency>

<groupId>org.apache.pig</groupId>

<artifactId>pig</artifactId>

<version>0.8.0</version>

<scope>compile</scope>

</dependency>

Add (under <dependencies> tag)

<dependency>

<groupId>org.easymock</groupId>

<artifactId>easymock</artifactId>

<version>2.5</version>

</dependency>

<dependency>

<groupId>org.apache.pig</groupId>

<artifactId>pig</artifactId>

<version>0.9.2</version>

</dependency>

<dependency>

<groupId>org.apache.pig</groupId>

<artifactId>piggybank</artifactId>

<version>1.0</version>

</dependency>

Remove (under <dependencies> tag) (this is a duplicate entry)

<dependency>

<groupId>org.apache.hadoop</groupId>

<artifactId>hadoop-core</artifactId>

<version>0.20.2</version>

<scope>compile</scope>

</dependency>

Modify (<dependency>/<log4j>/version) : 1.2.15 -> 1.2.16

Modify (<dependency>/<core-kafka_2.8.0>/groupid) : kafka -> org.apache.kafka

Modify (<dependency>/<core-kafka_2.8.0>/artifactid) : core-kafka_2.8.0 -> core

• Install piggybank.jar since its not available in the maven repository.

Command : mvn install:install-file -Dfile=./contrib/hadoop-producer/lib/piggybank.jar -DgroupId=org.apache.pig -DartifactId=piggybank -Dversion=1.0 -Dpackaging=jar

• Build/Install

Command : mvn clean -Dmaven.test.skip install (To skip runnings tests in case there are test failures)

Or

Command : mvn clean install (to run all the tests)