Ray3D: ray-based 3D human pose estimation for monocular ...

1 Ray3D: ray-based 3D human pose estimation for monocular absolute 3D. localization Yu Zhan Fenghai Li Renliang Weng Wongun Choi Aibee Inc. Beijing Technology and Business University Aibee Inc. Aibee Inc. [ ] 27 Oct 2022. Abstract In this paper, we propose a novel monocular ray-based S! S" = 2S! 3D (Ray3D) absolute human pose estimation with cali- ! Ground plane brated camera. Accurate and generalizable absolute 3D " = 2 ! Ground plane human pose estimation from monocular 2D pose input is Z! Z" = 2Z! an ill-posed problem.

2 To address this challenge, we con- vert the input from pixel space to 3D normalized rays. (a). f! f" = 2 ! This conversion makes our approach robust to camera in- trinsic parameter changes. To deal with the in-the-wild ! " = 1. camera extrinsic parameter variations, Ray3D explicitly takes the camera extrinsic parameters as an input and Z! jointly models the distribution between the 3D pose rays Z" = 2 Z! and camera extrinsic parameters. This novel network de- (b). sign is the key to the outstanding generalizability of Ray3D Figure 1.

3 As shown in (a), if both the body size and the distance approach. To have a comprehensive understanding of to camera are scaled up by twice, the projected 2D keypoints lo- how the camera intrinsic and extrinsic parameter varia- cations remain the same. Same phenomenon is observed in (b), tions affect the accuracy of absolute 3D key-point local- where both the focal length and 3D distance are doubled. Z1 and ization, we conduct in-depth systematic experiments on Z2 refer to the distance from the person to the camera, H1 and H2.

4 Three single person 3D benchmarks as well as one syn- represent the height of camera from the ground plane. S1 and S2. are the scale of the person. f1 and f2 represent focal length of the thetic benchmark. These experiments demonstrate that our camera. method significantly outperforms existing state-of-the-art models. Our code and the synthetic dataset are available than the root-relative 3D poses in the real-world applica- at tions. For instance, unmanned store requires to detect the merchandise picked up by the customer, which relies on ac- curate hand localization in world coordinate system.

5 1. Introduction A key-point's 2D pixel location is jointly determined by the scale of person's body figure, camera intrinsic param- Accurate monocular 3D human pose estimation has eters, camera extrinsic parameters and 3D position in the found its wide applications in augmented reality [26], world coordinate system. These factors introduce ambigui- human -object interaction [7], and video action recogni- ties for 3D pose estimation . For instance, as shown in Fig- tion [45]. While the problem has been extensively studied in ure 1 (a), if both the body size and the distance to camera recent years, it's a well-known ill-posed problem [25] with are scaled up by twice, the projected 2D key-points loca- limited generalization capability.

6 The problem becomes tions remain the same. Similarly, if both the focal length even more difficult when absolute 3D human pose estima- and 3D distance are doubled, the 2D key-points keep the tion in a metric space is required, as knowing exactly where same, as illustrated in Figure 1 (b). Typically, there are more a human joint is in the World Coordinate System (WCS) is than one configuration of 3D key-points that can generate much more challenging than estimating the relative 3D off- the same observation of 2D key-points in the image plane.

7 Set of that joint from a reference point. While being more Thus, naively learning a model to map from 2D pixel loca- challenging, knowing absolute 3D poses is more desirable tions to 3D world locations is arguably prone to failure. 1. To resolve these ambiguities, a number of monocular 3D pal point, camera pitch angle, camera height, camera yaw human estimation approaches have been proposed [5,11,31, angle, body figure size variations on synthetic dataset. Fur- 34, 49, 52]. These methods can be mainly categorized into thermore, we evaluate generalization capability of these ap- two groups, , lifting methods and image based meth- proaches on three single person benchmarks.

8 Ods. Lifting methods [3, 9, 12, 27, 34, 51, 55] take the 2D To summarize, the proposed method makes the follow- human poses as input and lift the 2D pose to 3D pose. A ing contributions, few lifting methods normalize the input according to image We convert the input space from 2D pixel space to 3D. resolution [34], and camera principal point [5]. While these rays in a normalized coordinate system. This simple normalization schemes improve the generalization ability to design effectively normalizes away the variations in- some extent, they fail to fully resolve the ambiguity due to troduced by the camera intrinsic parameter changes as variation in camera intrinsic parameters.

9 On the other hand, well as the camera pitch angle changes. image based approaches [2, 11, 18, 22, 24, 31, 48, 54] esti- We present a novel and simple network which learns a mate the 3D root position based on the prior about the body camera embedding using the camera extrinsic parame- size. In contrast, [31, 52] rely on image-based human depth ters, and jointly models the distribution of camera ex- estimation for absolute root-keypoint localization. The is- trinsic parameters and 3D rays. sue with these learning-based depth estimation approaches We provide a comprehensive and systematic bench- is lack of sufficient training data with viewpoint variations.

10 Marking of existing 3D approaches in terms of robust- For instance, the model trained with front-view viewpoint ness against camera pose variations, as well as cross- may not generalize well to cameras with large pitch value. dataset generalization. Moreover, they fail to fully address the aforementioned am- Experiments on three real benchmark datasets and one biguities. synthetic dataset clearly demonstrates the advantages To address the challenges more effectively, we propose of our Ray3D approach. our Ray3D method. Firstly, in order to have an intrinsic- parameter-invariant representation, we convert the 2D key- 2.