Apache Http-Client5 基本使用

Posted on Edited on In , Disqus:

Apache HttpComponents项目进行Http API使用及I/O Reactor

Apache http client 5.0 简介

Apache http client 5.0支持了HTTP/2的大部分特性，并对之前版本进行了一些列优化，具体如下：

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
Notable changes and features included in the 5.0 series are:

* Support for the HTTP/2 protocol and conformance to requirements and
  recommendations of the latest HTTP/2 protocol specification documents
  (RFC 7540, RFC 7541.)
  Supported features:
    ** HPACK header compression
    ** Stream multiplexing (client and server)
    ** Flow control
    ** Response push
    ** Message trailers
    ** Expect-continue handshake
    ** Connection validation (ping)
    ** Application-layer protocol negotiation (ALPN)
    ** TLS 1.2 security features
* Improved conformance to requirements and recommendations of the latest HTTP/1.1 protocol
  specification documents (RFC 7230, RFC 7231.)
* New connection pool implementation with lax connection limit guarantees and better
  performance under higher concurrency due to absence of a global pool lock.
* Support for Reactive Streams API [http://www.reactive-streams.org/]
* Package name space changed to 'org.apache.hc.client5'.
* Maven group id changed to 'org.apache.httpcomponents.client5'.

HttpClient 5.0 releases can be co-located with earlier major versions on the same classpath
due to the change in package names and Maven module coordinates.

Apache HttpComponents由HttpComponentsCore、HttpComponentsClient、HttpComponentsAysyncClient等部分组成。

maven dependence:

1
2
3
4
5
6
<dependency>
    <groupId>org.apache.httpcomponents.client5</groupId>
    <artifactId>httpclient5</artifactId>
    <version>5.0</version>
</dependency>

基本使用用法

表单登录获取cookie

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
public String admin_userName="admin";
public String admin_userPwd="adminX";
String formUserFiledName ="j_username";
String formPwdFiledName="j_password";
BasicCookieStore cookieStore = new BasicCookieStore();

//调用该方法注意对 CloseableHttpClient 在调用完成后执行 close()方法 || 可以复用
public CloseableHttpClient buildCookieHttpClient(String loginUrl) throws IOException, URISyntaxException {
     CloseableHttpClient httpClient = HttpClients.custom()
            .setDefaultCookieStore(cookieStore).build();

        //step1~2 cookie and login
        initCookie(httpClient, cookieStore,loginUrl);
        int statusCode = loginRanger(httpClient, cookieStore, loginUrl);
        if (statusCode != 200) {
            // login failed
            log.warn(" ranger-admin login failed ,status_code:{}", statusCode);
            throw new RuntimeException("login ranger-admin failed.");
        }
    return httpClient;
}
    private int loginRanger(CloseableHttpClient httpClient,BasicCookieStore cookieStore,String rangerLoginUrl) throws URISyntaxException, IOException {
    log.info("==> loginRanger userName:{}",admin_userName);
     ClassicHttpRequest login = ClassicRequestBuilder.post()
            .setUri(new URI(rangerLoginUrl))
            .addParameter(formUserFiledName, admin_userName)
            .addParameter(formPwdFiledName, admin_userPwd)
            .build();
    try (final CloseableHttpResponse loginResponse = httpClient.execute(login)) {
        final HttpEntity entity = loginResponse.getEntity();
        log.debug("Login form get: " + loginResponse.getCode() + " " + loginResponse.getReasonPhrase());
        EntityUtils.consume(entity);

        log.debug("====> Post login cookies:");
        final List<Cookie> cookies = cookieStore.getCookies();
        if (cookies.isEmpty()) {
            log.debug("===> Cookie None");
        } else {
            for (int i = 0; i < cookies.size(); i++) {
                log.debug("- " + cookies.get(i).toString());
            }
        }
        return loginResponse.getCode();
    }
}

 private void initCookie(CloseableHttpClient httpClient, BasicCookieStore cookieStore,String hostUrl)throws IOException {
     HttpGet httpget = new HttpGet(hostUrl);
    try (final CloseableHttpResponse response1 = httpClient.execute(httpget)) {
        final HttpEntity entity = response1.getEntity();
        log.debug("Login form get: " + response1.getCode() + " " + response1.getReasonPhrase());
        EntityUtils.consume(entity);
        log.debug("Initial set of cookies:");
        final List<Cookie> cookies = cookieStore.getCookies();
        if (cookies.isEmpty()) {
            log.warn(" get user cookies is None");
        } else {
            for (int i = 0; i < cookies.size(); i++) {
                log.debug("get user cookies - " + cookies.get(i).toString());
            }
        }
    }
}

官方Demo Apache-Client ClientFormLogin.java

POST request使用示例

POST request使用示例

1
2
3
4
5
6
7
8
9
10
11
12
13
 //默认的header
List<Header> defaultHeaderArr = Arrays.asList(new BasicHeader("Content-Type", "application/json"),
            new BasicHeader("Accept-Encoding", " gzip, deflate"),
            new BasicHeader("Accept", "application/json, text/javascript, */*; q=0.01"),
            new BasicHeader("Connection", "keep-alive"),
            new BasicHeader("X-Requested-With", "XMLHttpRequest"));

String jsonStr ="{}"
HttpPost request = new HttpPost(url);
headerList.stream().forEach(request::addHeader);
request.setEntity(new StringEntity(jsonStr, ContentType.APPLICATION_JSON));
httpClient.execute(request);

Get request 使用示例

Get request 使用demo:

1
2
3
4
5
6
7
8
9
10
11
12
//build get request with paramer
String url =hostUrl + "/service/plugins/policies/service" + "/" + serviceId;
HttpGet getRequest  = new HttpGet(url);
getRequest.addHeader(HttpHeaders.ACCEPT,"application/json, text/javascript, */*; q=0.01");
URIBuilder uriBuilder = new URIBuilder(getRequest.getUri());
URI uri = uriBuilder
        .addParameter("page", "0")
        .addParameter("pageSize", "2000")
        .addParameter("startIndex", "0")
        .build();
getRequest.setUri(uri);
httpClient.execute(getRequest);

官方Demo GetQuickStart.java

拦截器特性

可以通过给client request增加 interceptor可以增加request id和异常拦截等功能

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
public static void main(final String[] args) throws Exception {
        try (final CloseableHttpClient httpclient = HttpClients.custom()
                // Add a simple request ID to each outgoing request
                .addRequestInterceptorFirst(new HttpRequestInterceptor() {
                    private final AtomicLong count = new AtomicLong(0);
                    @Override
                    public void process(final HttpRequest request,final EntityDetails entity,
                            final HttpContext context) throws HttpException, IOException {
                        request.setHeader("request-id", Long.toString(count.incrementAndGet()));
                    }
                })
                // Simulate a 404 response for some requests without passing //the message down to the backend
                .addExecInterceptorAfter(ChainElement.PROTOCOL.name(), "custom", new ExecChainHandler() {
                    @Override
                    public ClassicHttpResponse execute(
                            final ClassicHttpRequest request,
                            final ExecChain.Scope scope,
                            final ExecChain chain) throws IOException, HttpException {
                        final Header idHeader = request.getFirstHeader("request-id");
                        if (idHeader != null && "13".equalsIgnoreCase(idHeader.getValue())) {
                            final ClassicHttpResponse response = new BasicClassicHttpResponse(HttpStatus.SC_NOT_FOUND, "Oppsie");
                            response.setEntity(new StringEntity("bad luck", ContentType.TEXT_PLAIN));
                            return response;
                        } else {
                            return chain.proceed(request, scope);
                        }
                    }
                })
                .build()) {
            for (int i = 0; i < 20; i++) {
                final HttpGet httpget = new HttpGet("http://httpbin.org/get");
                System.out.println("Executing request " + httpget.getMethod() + " " + httpget.getUri());
                try (final CloseableHttpResponse response = httpclient.execute(httpget)) {
                    System.out.println(response.getCode() + " " + response.getReasonPhrase());
                    System.out.println(EntityUtils.toString(response.getEntity()));
                }
            }
        }
    }

参照官方Demo ClientInterceptors.java

线程池

使用PoolingHttpClientConnectionManager可以实现多线程并发访问，在并发场景下推荐使用异步模式。

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
// Create an HttpClient with the PoolingHttpClientConnectionManager.
 // This connection manager must be used if more than one thread will
 // be using the HttpClient.
 final PoolingHttpClientConnectionManager cm = new PoolingHttpClientConnectionManager();
 cm.setMaxTotal(100);
 try (final CloseableHttpClient httpclient = HttpClients.custom()
         .setConnectionManager(cm)
         .build()) {
     // create an array of URIs to perform GETs on
     final String[] urisToGet = {
             "http://hc.apache.org/",
             "http://hc.apache.org/httpcomponents-core-ga/",
             "http://hc.apache.org/httpcomponents-client-ga/",
     };
     // create a thread for each URI
     final GetThread[] threads = new GetThread[urisToGet.length];
     for (int i = 0; i < threads.length; i++) {
         final HttpGet httpget = new HttpGet(urisToGet[i]);
         threads[i] = new GetThread(httpclient, httpget, i + 1);
     }
     // start the threads
     for (final GetThread thread : threads) {
         thread.start();
     }
     // join the threads
     for (final GetThread thread : threads) {
         thread.join();
     }

发送多个部分的参数

使用MultipartEntityBuilder可以实现发送多个不同的种类的请求参数：

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
try (final CloseableHttpClient httpclient = HttpClients.createDefault()) {
    final HttpPost httppost = new HttpPost("http://localhost:8080" +
            "/servlets-examples/servlet/RequestInfoExample");
    final FileBody bin = new FileBody(new File(args[0]));
    final StringBody comment = new StringBody("A binary file of some kind", ContentType.TEXT_PLAIN);
    final HttpEntity reqEntity = MultipartEntityBuilder.create()
            .addPart("bin", bin)
            .addPart("comment", comment)
            .build();
    httppost.setEntity(reqEntity);
    System.out.println("executing request " + httppost);
    try (final CloseableHttpResponse response = httpclient.execute(httppost)) {
        System.out.println(response);
        final HttpEntity resEntity = response.getEntity();
        if (resEntity != null) {
            System.out.println("Response content length: " + resEntity.getContentLength());
        }
        EntityUtils.consume(resEntity);
    }
}

异步请求

基于NIO的HttpAsyncClients 很好的实现了异步请求。

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
final IOReactorConfig ioReactorConfig = IOReactorConfig.custom()
               .setSoTimeout(Timeout.ofSeconds(5))
               .build();
       final CloseableHttpAsyncClient client = HttpAsyncClients.custom()
               .setIOReactorConfig(ioReactorConfig)
               .build();
       client.start();
       final String requestUri = "http://httpbin.org/post";
       final AsyncRequestProducer requestProducer = AsyncRequestBuilder.post(requestUri)
               .setEntity(new StringAsyncEntityProducer("some stuff", ContentType.TEXT_PLAIN))
               .build();
       final Future<SimpleHttpResponse> future = client.execute(requestProducer, SimpleResponseConsumer.create(),
               new FutureCallback<SimpleHttpResponse>() {
                   @Override
                   public void completed(final SimpleHttpResponse response) {
                       System.out.println(requestUri + "->" + response.getCode());
                       System.out.println(response.getBody());
                   }
                   @Override
                   public void failed(final Exception ex) {
                       System.out.println(requestUri + "->" + ex);
                   }
                   @Override
                   public void cancelled() {
                       System.out.println(requestUri + " cancelled");
                   }
               });
       try {
           future.get();
       } catch (InterruptedException e) {
           e.printStackTrace();
       } catch (ExecutionException e) {
           e.printStackTrace();
       }
       System.out.println("Shutting down");
       client.close(CloseMode.GRACEFUL);

http-client5日志配置

http-client5使用了slf4j输出日志，如果需要配置为log4j2需要进行一下配置：

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
<dependency>
            <groupId>org.apache.logging.log4j</groupId>
            <artifactId>log4j-core</artifactId>
            <version>2.13.2</version>
        </dependency>
        <dependency>
            <groupId>org.apache.logging.log4j</groupId>
            <artifactId>log4j-api</artifactId>
            <version>2.13.2</version>
        </dependency>
        <dependency>
            <groupId>org.apache.logging.log4j</groupId>
            <artifactId>log4j-slf4j-impl</artifactId>
            <version>2.13.2</version>
        </dependency>

log4j2.xml配置：

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
<Configuration>
    <Appenders>
        <Console name="STDOUT" target="SYSTEM_OUT" >
            <PatternLayout pattern="%d{HH:mm:ss.SSS} [%thread] %-5level %logger{36}.%M\(%line\) - %msg%n" />
        </Console>

    </Appenders>
    <Loggers>
        <Logger name="org.apache.hc.client5.http" level="INFO">
            <AppenderRef ref="Console"/>
        </Logger>
        <Logger name="org.apache.hc.client5.http.wire" level="DEBUG">
            <AppenderRef ref="Console"/>
        </Logger>
        <Root level="DEBUG">
            <AppenderRef ref="STDOUT" />
        </Root>
    </Loggers>
</Configuration>

官方配置说明 logger guide

HttpComponents的NIO实现——I/O reactor

HttpComponents Core包含了2种I/O实现，分别是是BIO和NIO。（参照官方说明） NIO是基于Doug Lea原子模型的实现的NIO(netty 已经采用Reactor pattern很多年了～～)，关于Doug Lea的这篇文章Scalable IO in Java - Doug Lea 基本原子线程模型（Doug Lea)： images

多线程的原子线程模型（Doug Lea)： images

参考

apache HttpComponents httpcomponents-jackson