Posted on   |   In

测试环境为eclipse-4.5.1，内置已经支持maven，本机已经配置伪分布式hadoop-2.7.1环境，并且已经启动start-dfs.sh和start-yarn.sh。

注意：以下所有配置中的hadoop-username，需要修改成实际测试环境中的用户名。

新建maven项目

pom.xml文件内容如下：

<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"    xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">    <modelVersion>4.0.0</modelVersion>    <groupId>com.example</groupId>    <artifactId>hadoop</artifactId>    <version>0.0.1</version>    <name>hadoop mapreduce</name>    <description>test hadoop-2.7.1 mapreduce.</description>    <build>        <plugins>            <plugin>                <groupId>org.apache.maven.plugins</groupId>                <artifactId>maven-compiler-plugin</artifactId>                <version>3.5</version>                <configuration>                    <source>1.8</source>                    <target>1.8</target>                </configuration>            </plugin>            <plugin>                <groupId>org.codehaus.mojo</groupId>                <artifactId>exec-maven-plugin</artifactId>                <version>1.2.1</version>            </plugin>        </plugins>    </build>    <properties>        <hadoop.version>2.7.1</hadoop.version>    </properties>    <dependencies>        <dependency>            <groupId>org.apache.hadoop</groupId>            <artifactId>hadoop-hdfs</artifactId>            <version>${hadoop.version}</version>        </dependency>        <dependency>            <groupId>org.apache.hadoop</groupId>            <artifactId>hadoop-common</artifactId>            <version>${hadoop.version}</version>        </dependency>        <dependency>            <groupId>org.apache.hadoop</groupId>            <artifactId>hadoop-mapreduce-client-common</artifactId>            <version>${hadoop.version}</version>        </dependency>    </dependencies></project>
Read more »

Posted on   |   In

一句话介绍Homebrew：

The missing package manager for macOS

Homebrew使用示例

Homebrew的可执行命令是brew，其基本使用方法如下：

查找软件包

 brew search $FORMULA

安装软件包

 brew install $FORMULA

列出已安装的软件包

 brew list

编辑安装文件

 brew edit wget # 使用 $EDITOR 编辑

删除软件包

 brew remove $FORMULA

查看软件包信息

 brew info $FORMULA

列出软件包的依赖关系

 brew deps --installed --tree # 查看已安装的包的依赖，树形显示

更新brew

 brew update

列出过时的软件包

 brew outdated

更新过时的软件包

 brew upgrade             # 更新所有的包 brew upgrade $FORMULA    # 更新指定的包

清理旧版本

 brew cleanup             # 清理所有包的旧版本 brew cleanup $FORMULA    # 清理指定包的旧版本 brew cleanup -n          # 查看可清理的旧版本包，不执行实际操作

锁定不想更新的包

 brew pin $FORMULA      # 锁定某个包 brew unpin $FORMULA    # 取消锁定

从/usr/local快速删除链接

 brew unlink $FORMULA

恢复链接

 brew link $FORMULA

激活指定版本

 brew switch $FORMULA $VERSION

不使用formulae手工安装

 ./configure --prefix=/usr/local/Cellar/foo/1.2 && make && make install && brew link foo

下载安装的缓存目录

 brew --cache

~/Library/Caches/Homebrew

安装提前下载的安装包

将下载的安装包放到~/Library/Caches/Homebrew目录，并注意下载的文件名要符合Homebrew的要求。

使用http代理安装软件

 http_proxy=http://<proxyhost>:<proxyport> brew install $FORMULA http_proxy=http://<user>:<password>@<proxyhost>:<proxyport> brew install $FORMULA

References

  1. homebrew
  2. The missing package manager for macOS
  3. keeping homebrew date
  4. FAQ of homebrew

Posted on   |   In

以下操作命令是在Mac OS上进行的，linux上略有不同，比如${JAVA_HOME}设置。

ssh rsa公钥设置

设置本地ssh公钥私钥，使ssh可以无密码登录本地，以下命令中出现提示，则按提示回答。

 ssh-keygen -t rsa -N "" -f ~/.ssh/id_rsa# ssh-copy-id localhost cat ~/.ssh/id_rsa.pub >> ~/.ssh/authorized_keys ssh localhost

下载安装预编译版本hadoop-2.7.1

 wget https://archive.apache.org/dist/hadoop/core/hadoop-2.7.1/hadoop-2.7.1.tar.gz tar zxvf hadoop-2.7.1.tar.gz mkdir -p ~/Applications mv hadoop-2.7.1 ~/Applications/hadoop

配置hadoop-2.7.1运行环境

添加以下内容到~/.bash_profile文件如下(注意linux中JAVA_HOME配置与此不同)，Mac OS还可以参考此文：

export JAVA_HOME=$(/usr/libexec/java_home)export HADOOP_HOME=~/Applications/hadoopexport HADOOP_PREFIX=$HADOOP_HOMEexport PATH=$PATH:$HADOOP_HOME/binexport PATH=$PATH:$HADOOP_HOME/sbinexport HADOOP_MAPRED_HOME=$HADOOP_HOMEexport HADOOP_COMMON_HOME=$HADOOP_HOMEexport HADOOP_HDFS_HOME=$HADOOP_HOMEexport YARN_HOME=$HADOOP_HOMEexport HADOOP_COMMON_LIB_NATIVE_DIR=$HADOOP_HOME/lib/nativeexport HADOOP_OPTS="-Djava.library.path=$HADOOP_COMMON_LIB_NATIVE_DIR"
Read more »

Posted on   |   In

Mac OS 10.9下运行官网直接下载的hadoop-2.7.1包中的start-dfs.sh命令后，出现如下提示信息：

16/01/26 15:00:41 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable

官方文档说明native hadoop library不支持Mac OS X平台，不过可以通过homebrew安装环境后支持，如下：

The native hadoop library is supported on *nix platforms only. The library does not to work with Cygwin or the Mac OS X platform.

安装hadoop-2.7.1编译环境

 brew update brew install homebrew/dupes/zlib brew install cmake brew install bzip2 brew install lz4 brew install snappy brew install openssl

安装hadoop-2.7.1对应的protobuf-2.5.0版本。

 brew install https://raw.githubusercontent.com/Homebrew/homebrew-versions/master/protobuf250.rb protoc --versionlibprotoc 2.5.0
Read more »

Posted on   |   In

网上java8新特性详解这篇翻译翻得很好，可以结合以下英文原文内容学习java8新特性。

Modern Java - A Guide to Java 8

“Java is still not dead and people are starting to figure that out.”

Welcome to @winterbe's introduction to Java 8. This tutorial guides you step by step through all new language features. Backed by short and simple code samples you'll learn how to use default interface methods, lambda expressions, method references and repeatable annotations. At the end of the article you'll be familiar with the most recent API changes like streams, functional interfaces, map extensions and the new Date API. No walls of text, just a bunch of commented code snippets. Enjoy!

Table of Contents

Default Methods for Interfaces

Java 8 enables us to add non-abstract method implementations to interfaces by utilizing the default keyword. This feature is also known as virtual extension methods.

Here is our first example:

interface Formula {    double calculate(int a);    default double sqrt(int a) {        return Math.sqrt(a);    }}

Besides the abstract method calculate the interface Formula also defines the default method sqrt. Concrete classes only have to implement the abstract method calculate. The default method sqrt can be used out of the box.

Formula formula = new Formula() {    @Override    public double calculate(int a) {        return sqrt(a * 100);    }};formula.calculate(100);     // 100.0formula.sqrt(16);           // 4.0

The formula is implemented as an anonymous object. The code is quite verbose: 6 lines of code for such a simple calculation of sqrt(a * 100). As we'll see in the next section, there's a much nicer way of implementing single method objects in Java 8.

Read more »