Apache Atlas Compilation and installation records

1、 Environmental preparation

atlas Now it is a very popular data management tool in the market , however atlas After downloading, it is a source code project , Cannot be used directly , So compile the source code ( It is the process from source program to target program, including code generation 、 Code integration 、 Syntax analysis 、 Lexical analysis 、 Dependent Download ) Before installation and use .

linux

However, in the compilation process, the corresponding dependent packages should be downloaded through the network , Therefore, the server in the Intranet environment cannot be used for the time being , So I use the cloud server of Tencent cloud .

JDK 8 install

• Download address https://www.oracle.com/java/technologies/downloads/#java8

• download ：jdk-8u311-linux-x64.rpm

• Transfer the installation file to the ECS directory through the pagoda panel

• Use root jurisdiction

su root 

• Execute installation statement

rpm -ivh jdk-8u311-linux-x64.rpm

• verification

java -version

java version “1.8.0_311”
Java SE Runtime Environment (build 1.8.0_311-b11)
Java HotSpot 64-Bit Server VM (build 25.311-b11, mixed mode)

• Configure environment variables

vim /etc/profile

export JAVA_HOME=/usr/java/jdk1.8.0_311-amd64
export JRE_HOME=$JAVA_HOME/jre
export CLASSPATH=$JAVA_HOME/lib:$JRE_HOME/lib:$CLASSPATH
export PATH=$JAVA_HOME/bin:$JRE_HOME/bin:$PATH

source /etc/profile

verification

echo $JAVA_HOME

/usr/java/jdk1.8.0_311-amd64

maven install

• download Official website address : http://maven.apache.org/download.cgi
• Upload the installation package to ECs
• tar -zxvf apache-maven-3.6.3-bin.tar.gz
• Configure environment variables

vi /etc/profile

export MAVEN_HOME=/tmp/apache-maven-3.6.3
export PATH=$MAVEN_HOME/bin:$JAVA_HOME/bin:$JRE_HOME/bin:$PATH

• Refresh environment variable

source /etc/profile

• Check version

mvn -v 

2、 compile

The download of compiled files through foreign images is very slow , Let's add a domestic download image , Used to speed up compilation ( Download dependency package ) Speed

vim /tmp/apache-maven-3.6.3/conf/settings.xml

stay mirrors Add a domestic image

<mirror>
    <id>alimaven</id>
    <name>aliyun maven</name>
    <url>http://maven.aliyun.com/nexus/content/groups/public/</url>
    <mirrorOf>central</mirrorOf>
</mirror>

• maven When calling the configuration file, the priority is to call /root/.m2/( Hide directory ) Below
• establish /root/.m2 A directory and then copy the configuration file

[conf]# cp settings.xml /root/.m2/

download atlas

• Download address https://atlas.apache.org/#/Downloads
  

decompression

• tar -zxvf apache-atlas-2.2.0-sources.tar.gz

• Resolve version conflicts , take zookeeper and hbase And other dependent versions are modified to be consistent with their own environment ( Or compatible ) Version of . At the same time, considering the need to use the domestic image website to download the image file ,hbase and solr The version should be in https://mirrors.tuna.tsinghua.edu.cn/apache/ in .

• such as hbase We can now choose 2.3.7 edition , If you choose 2.3.3, You cannot find the path
  
  

<hbase.version>2.3.7</hbase.version>
<solr.version>8.11.1</solr.version>

• stay distro modular （ Sub project ） Medium pom File modification hbase and solr The download path of is the download website of Tsinghua University , Download much faster

<hbase.tar>http://mirrors.tuna.tsinghua.edu.cn/apache/hbase/${hbase.version}/hbase-${hbase.version}-bin.tar.gz</hbase.tar>
<solr.tar>http://mirrors.tuna.tsinghua.edu.cn/apache/lucene/solr/${solr.version}/solr-${solr.version}.tgz</solr.tar>

compile /usr/apps/apache-atlas-sources-2.0.0 file

• First, increase the initial heap memory of the compiler , Is increasing the maximum heap memory , Because the files are bigger , Use more memory

[apache-atlas-sources-2.2.0]# export MAVEN_OPTS="-Xms2g -Xmx2g"

• use maven Clear the test program package, Compile a release that contains hbase and solr The program

[apache-atlas-sources-2.2.0]# mvn clean -DskipTests package -Pdist,embedded-hbase-solr

3、 install

After compilation , Download the compiled from ECs apache-atlas-2.2.0-server.tar.gz file .
After downloading, upload the compiled file to the local server .

Perform decompression and installation

tar zxvf apache-atlas-2.2.0-server.tar.gz

Get into apache-atlas-2.2.0 Catalog

Modify the configuration file

vim conf/atlas-env.sh

Set up java environment variable

JAVA_HOME: export JAVA_HOME=/opt/bigdata/jdk1.8.0_311

I use atlas Self contained hbase and solr So it's going to be set to true

export MANAGE_LOCAL_HBASE=true
export MANAGE_LOCAL_SOLR=true

About atlas-application.properties, I stepped on the pit several times , Modify to modify , Finally, when the operation is successful, there are only one more configuration below 2181 Port number , You can also use the default when watching others online , It should work .

atlas.graph.storage.hostname=localhost:2181

The key here is to use Atlas Self contained hbase and solr, That's what we use hbase Self contained zk, therefore solr Of zk Don't change the address , Otherwise, the report will be wrong .

Do not start after configuration atlas, Abnormal transactions , as a result of hbase and solr At the same time to start , and solr It's using hbase Inside zk, So start it first hbase, Restart solr, Finally, start atlas

start-up hbase

[root@hive2 apache-atlas-2.2.0]# hbase/bin/start-hbase.sh

start-up solr

• Note that there zk It's using localhost:2181
• -c: Create a table -z: Turn on zookeeper,hbase Instead of opening , -p:solr The port of -force: Force on

[root@hive2 apache-atlas-2.2.0]# solr/bin/solr start -c -z localhost:2181 -p 8983 -force

• Create index nodes , Otherwise start will report an error

solr/bin/solr  create -c fulltext_index -force -d conf/solr/
solr/bin/solr  create -c edge_index -force -d conf/solr/
solr/bin/solr  create -c vertex_index -force -d conf/solr/

• Access address localhost:8983 ( If hosts The file configuration is invalid. You can install the server directly ip Address substitution )
  

start-up atlas

[root@hive2 apache-atlas-2.2.0]# bin/atlas_start.py

• Access address localhost:21000
• The initial user name and password are admin
  

Shutdown command

 stop it atlas
[root@hive2 apache-atlas-2.2.0]# bin/atlas_stop.py
 stop it hbase
[root@hive2 apache-atlas-2.2.0]# hbase/bin/stop-hbase.sh
 stop it solr
[root@hive2 apache-atlas-2.2.0]# solr/bin/solr stop

4、 Step on the pit

4.1 abnormal No goals have been specified for this build

[ERROR] No goals have been specified for this build. You must specify a valid lifecycle phase or a goal in the format <plugin-prefix>:<goal> or <plugin-group-id>:<plugin-artifact-id>[:<plugin-version>]:<goal>. Available lifecycle phases are: validate, initialize, generate-sources, process-sources, generate-resources, process-resources, compile, process-classes, generate-test-sources, process-test-sources, generate-test-resources, process-test-resources, test-compile, process-test-classes, test, prepare-package, package, pre-integration-test, integration-test, post-integration-test, verify, install, deploy, pre-clean, clean, post-clean, pre-site, site, post-site, site-deploy. -> [Help 1]
org.apache.maven.lifecycle.NoGoalSpecifiedException: No goals have been specified for this build. You must specify a valid lifecycle phase or a goal in the format <plugin-prefix>:<goal> or <plugin-group-id>:<plugin-artifact-id>[:<plugin-version>]:<goal>. Available lifecycle phases are: validate, initialize, generate-sources, process-sources, generate-resources, process-resources, compile, process-classes, generate-test-sources, process-test-sources, generate-test-resources, process-test-resources, test-compile, process-test-classes, test, prepare-package, package, pre-integration-test, integration-test, post-integration-test, verify, install, deploy, pre-clean, clean, post-clean, pre-site, site, post-site, site-deploy.
    at org.apache.maven.lifecycle.internal.LifecycleStarter.execute (LifecycleStarter.java:97)
    at org.apache.maven.DefaultMaven.doExecute (DefaultMaven.java:305)
    at org.apache.maven.DefaultMaven.doExecute (DefaultMaven.java:192)
    at org.apache.maven.DefaultMaven.execute (DefaultMaven.java:105)
    at org.apache.maven.cli.MavenCli.execute (MavenCli.java:957)
    at org.apache.maven.cli.MavenCli.doMain (MavenCli.java:289)
    at org.apache.maven.cli.MavenCli.main (MavenCli.java:193)
    at sun.reflect.NativeMethodAccessorImpl.invoke0 (Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke (NativeMethodAccessorImpl.java:62)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke (DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke (Method.java:498)
    at org.codehaus.plexus.classworlds.launcher.Launcher.launchEnhanced (Launcher.java:282)
    at org.codehaus.plexus.classworlds.launcher.Launcher.launch (Launcher.java:225)
    at org.codehaus.plexus.classworlds.launcher.Launcher.mainWithExitCode (Launcher.java:406)
    at org.codehaus.plexus.classworlds.launcher.Launcher.main (Launcher.java:347)
[ERROR] 
[ERROR] 
[ERROR] For more information about the errors and possible solutions, please read the following articles:
[ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/NoGoalSpecifiedException

solve

If the above error occurs, you can try to use the following configuration

pom.xml  In file  <build>  The label should be configured ： <defaultGoal>compile</defaultGoal> 

4.2 Error creating assembly archive impala-hook

[INFO] Building tar: /tmp/apache-atlas-sources-2.2.0/distro/target/apache-atlas-2.2.0-hive-hook.tar.gz
[INFO] ------------------------------------------------------------------------
[INFO] BUILD FAILURE
[INFO] ------------------------------------------------------------------------
[INFO] Total time:  07:42 min
[INFO] Finished at: 2021-12-29T16:08:19+08:00
[INFO] ------------------------------------------------------------------------
[ERROR] Failed to execute goal org.apache.maven.plugins:maven-assembly-plugin:2.4.1:single (default) on project atlas-distro: Failed to create assembly: Error creating assembly archive impala-hook: You must set at least one file. -> [Help 1]
[ERROR] 
[ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch.
[ERROR] Re-run Maven using the -X switch to enable full debug logging.
[ERROR] 
[ERROR] For more information about the errors and possible solutions, please read the following articles:
[ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/MojoExecutionException

solve

This error report is revised by me many times pom Encountered after file , I probably located it because all-jar.xml Not in the file fileset Why , But I believe there is nothing wrong with the source code , therefore , I finally chose to reset the of the main project pom file , Modify only hbase and solr Version of , Nothing else moved . The final compilation is successful .

4.3 Service Unavailable 503

solve

There's an error , Don't spend time looking for anything IIS 了 , Let's take a look at logs/application.log Log files , Mostly because it didn't start successfully , Check some configurations , See if there is any hand slip to correct the mistake .